lightninglabs/neutrino

Privacy-Preserving Bitcoin Light Client

Neutrino: Privacy-Preserving Bitcoin Light Client

Neutrino is a Bitcoin light client written in Go and designed with mobile Lightning Network clients in mind. It uses a new proposal for compact block filters to minimize bandwidth and storage use on the client side, while attempting to preserve privacy and minimize processor load on full nodes serving light clients.

Mechanism of operation

The light client synchronizes only block headers and a chain of compact block filter headers specifying the correct filters for each block. Filters are loaded lazily and stored in the database upon request; blocks are loaded lazily and not saved.

Usage

The client is instantiated as an object using NewChainService and then started. Upon start, the client sets up its database and other relevant files and connects to the p2p network. At this point, it becomes possible to query the client.

Queries

There are various types of queries supported by the client. There are many ways to access the database, for example, to get block headers by height and hash; in addition, it's possible to get a full block from the network using GetBlockFromNetwork by hash. However, the most useful methods are specifically tailored to scan the blockchain for data relevant to a wallet or a smart contract platform such as a Lightning Network node like lnd. These are described below.

Rescan

Rescan allows a wallet to scan a chain for specific TXIDs, outputs, and addresses. A start and end block may be specified along with other options. If no end block is specified, the rescan continues until stopped. If no start block is specified, the rescan begins with the latest known block. While a rescan runs, it notifies the client of each connected and disconnected block; the notifications follow the btcjson format with the option to use any of the relevant notifications. It's important to note that "recvtx" and "redeemingtx" notifications are only sent when a transaction is confirmed, not when it enters the mempool; the client does not currently support accepting 0-confirmation transactions.

GetUtxo

GetUtxo allows a wallet or smart contract platform to check that a UTXO exists on the blockchain and has not been spent. It is highly recommended to specify a start block; otherwise, in the event that the UTXO doesn't exist on the blockchain, the client will download all the filters back to block 1 searching for it. The client scans from the tip of the chain backwards, stopping when it finds the UTXO having been either spent or created; if it finds neither, it keeps scanning backwards until it hits the specified start block or, if a start block isn't specified, the first block in the blockchain. It returns a SpendReport containing either a TxOut including the PkScript required to spend the output, or containing information about the spending transaction, spending input, and block height in which the spending transaction was seen.

Stopping the client

Calling Stop on the ChainService client allows the user to stop the client; the method doesn't return until the ChainService is cleanly shut down.

neutrino's Issues

Add support for Tor

We should add support for Tor; this is probably not hard as btcd, from which much of the connectivity code is taken, supports Tor.

How should empty filters be serialized?

I'm implementing the data structures and algorithms for bip157 and bip158 in NBitcoin. Using the test vectors referenced by the bip158 document i see that block height=0 (genesis) and block height=1 have an empty extended filter becuase they only have one transaction each without inputs (the coinbase tx). However those two empty filters are serialized differently, the first one is serialized as "00" and the second one is serialized as "" (empty string).

Are both correct? Should I check for filter.N > 0 before comparing their serialized representation?

PS: i'm not sure is this is the correct project where to make this question so, please let me know it. Thx

Why is the Index so big?

On the mailing list you say the "the total size of the index comes out to 6.9GB."

I was not able to fully understand your scheme, so I decided to scratch up my own simplified version of it: Client Side Filtering-- until mainnet nodes start to serve Neutrino. In my version, the index is 1.75GB, despite zero optimization, using some old, inefficient code of mine.

[Questions] Privacy, Mainnet usage

Did anyone evaluated the privacy properties of this client? If so, can you link it?

Do people use Neutrino on the mainnet? If I'm not mistaken the lnd daemon can use Neutrino as the back end currently. Is that right?

As a background, I'd like to change HiddenWallet's back end to Neutrino (HiddenWallet is a full block downloading SPV wallet) if I find its privacy properties sufficient.
I'm also thinking of implementing Neutrino from scratch in .NET, but I'm not sure that's a good approach. At such early stages competing implementations are just asking for trouble.

Unable to query bootstrapper BOLT-0010 DNS Seed

keep on getting this on Mac, sent a tx from faucet. received nothing so far.

lnd: 2017-10-14 21:54:12.334 [INF] DISC: Attempting to bootstrap with: BOLT-0010 DNS Seed: [nodes.lightning.directory]
lnd: 2017-10-14 21:54:12.993 [ERR] DISC: Unable to query bootstrapper BOLT-0010 DNS Seed: [nodes.lightning.directory]: lookup _nodes._tcp.nodes.lightning.directory on [2602:306:3028:ece0::1]:53: no such host 2017-10-14 21:54:12.993 [INF] DISC: Obtained 0 addrs to bootstrap network with
lnd: 2017-10-14 21:54:23.640 [INF] RPCS: [listchannels] fetched 0 channels from DB

Segmentation Fault

Testing v0.4.4 of lightning-app, which is packaged with:

lnd version 0.5.2-99-beta commit=queue/v1.0.1-17-g09059eacf06068aefe6e5b23a0276fdcd8021d86

I've encountered segmentation faults in Neutrino:

[17:22:35.400] [error] lnd Error: panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8f1978]

goroutine 168430 [running]:
github.com/lightninglabs/neutrino.rescan.func1.1()
    /home/travis/gocode/pkg/mod/github.com/lightninglabs/[email protected]/rescan.go:444 +0x1d8

[17:22:35.401] [error] lnd Error: created by time.goFunc
    /home/travis/go/src/time/sleep.go:172 +0x44

Same thing using a prerelease build of Zap-Desktop:

  zap:lnd [ERR]  panic: runtime error: invalid memory address or nil pointer dereference +3ms
  zap:lnd [ERR]  [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x152fada] +1ms
  zap:lnd [ERR]   +0ms
  zap:lnd [ERR]  goroutine 46479 [running]: +0ms
  zap:lnd [ERR]  github.com/lightninglabs/neutrino.rescan.func1.1() +1ms
  zap:lnd [ERR]      /Users/tom/.go/pkg/mod/github.com/lightninglabs/[email protected]/rescan.go:444 +0x1da +0ms
  zap:lnd [ERR]  created by time.goFunc +0ms
  zap:lnd [ERR]      /usr/local/opt/go/libexec/src/time/sleep.go:169 +0x44 +1ms
  zap:main [ERR]  Error
  zap:main ___Stack trace #1___
  zap:main Error
  zap:main     at Http2CallStream.call.on (/Users/jmow/Projects/bolt/bolt-desktop/node_modules/@grpc/grpc-js/build/src/call.js:52:41)
  zap:main     at Http2CallStream.emit (events.js:187:15)
  zap:main     at Http2CallStream.endCall (/Users/jmow/Projects/bolt/bolt-desktop/node_modules/@grpc/grpc-js/build/src/call-stream.js:58:18)
  zap:main     at ClientHttp2Stream.stream.on (/Users/jmow/Projects/bolt/bolt-desktop/node_modules/@grpc/grpc-js/build/src/call-stream.js:298:22)
  zap:main     at process._tickCallback (internal/process/next_tick.js:68:7) +52ms
  zap:main [DBG]  CHANNELGRAPH STATUS: [object Object]
  zap:main ___Inspected object #1___
  zap:main { code: 13,
  zap:main   details: '',
  zap:main   metadata: Metadata { internalRepr: Map {} } } +0ms
  zap:main [ERR]  Error
  zap:main ___Stack trace #1___
  zap:main Error
  zap:main     at Http2CallStream.call.on (/Users/jmow/Projects/bolt/bolt-desktop/node_modules/@grpc/grpc-js/build/src/call.js:52:41)
  zap:main     at Http2CallStream.emit (events.js:187:15)
  zap:main     at Http2CallStream.endCall (/Users/jmow/Projects/bolt/bolt-desktop/node_modules/@grpc/grpc-js/build/src/call-stream.js:58:18)
  zap:main     at ClientHttp2Stream.stream.on (/Users/jmow/Projects/bolt/bolt-desktop/node_modules/@grpc/grpc-js/build/src/call-stream.js:298:22)
  zap:main     at process._tickCallback (internal/process/next_tick.js:68:7) +1ms
  zap:main [INF]  INVOICE STATUS: [object Object]

No any peer

I am trying to get the latest block using neutrino. But no any peers are ok for neutrino.

I get : Disconnecting peer xx.xxx.xx.xx:8333 (outbound), cannot serve compact filters

here is my code :

db, err := walletdb.Create("bdb", "/tmp/wallet.db")
	if err != nil {
		log.Fatalf("unable to create db: %v", err)
		return
	}
	svc, err := neutrino.NewChainService(neutrino.Config{
		DataDir:     "/tmp/",
		Database:    db,
		ChainParams: chaincfg.MainNetParams,
	})
	

Writes to headerStore not ACID

Hi, I just took a look at the headerStore and noticed that you use appendRaw for writing headers atomically. Seems like the actual transaction to disk is not ACID though which could lead to corrupted writes to the store in some situations.
If that's a concern you could check out our writeaheadlog implementation which we came up with due to the need of writing all kinds of metadata rapidly and in an ACID way.
If you like it I could also create a PR.

Improve TX broadcast

It's worthwhile to try to improve transaction broadcast for both additional privacy (trickling to peers instead of sending to all) and propagation monitoring (which isn't possible to do with the current setup as we turn off transaction inv messages). We should investigate how to improve this without giving up privacy by turning the transaction inv messages on and off when we send a transaction; this may involve additional mechanisms for watching unconfirmed transactions such as a more detailed inv as suggested here.

lnd panic while syncing

Hi!
While syncing the Lighting Desktop app on Windows 10, I got the following error:

lnd Error: panic: Rollback failed: truncate C:\Users\Coco\AppData\Local\Lnd\data\testnet3\bitcoin\reg_filter_headers.bin: Access is denied. goroutine 50 [running]: github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*blockManager).handleHeadersMsg(0xc0424b88c0, 0xc044ed05e0) /Users/bryanvu/work/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blockmanager.go:1085 +0x1d3a github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*blockManager).blockHandler(0xc0424b88c0) /Users/bryanvu/work/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blockmanager.go:347 +0x434
lnd Error: created by github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*blockManager).Start /Users/bryanvu/work/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blockmanager.go:200 +0x11b

Feature request - progress endpoint

When a new node is started up, it takes approximately 25 minutes for the LND daemon to start up and listen on port 9735. Neutrino is awesome and is building filters at a rapid pace. Yet in a mobile environment, apart from training videos, documentation, there is no other way to provide a feedback or a progress bar as to the status of neutrino while it is trying to fully sync.

Thinking a progress endpoint would be great with at least two fields, 1) Caught up with block, 2) total blocks - something like 140000/1,4000,000. We could then use the data to post a progress bar.
This would be for the very first time the node is created, and it would be useful even subsequently, only that the syncing process is generally finished in under 30 seconds (depending on how the mobile node has been off line).

Stoping LND with Control C - many request \of checkpointed cfheaders

With #105 and #106, when pressing control+c to terminate LND while syncing. I got this:

2018-10-21 23:54:34.708 [DBG] BTCN: Received cfheaders from 159.203.125.125:18333 (outbound)
2018-10-21 23:54:34.709 [DBG] BTCN: Got cfheaders from height=19001 to height=20000, prev_hash=16b1dbce6d649dbd6bce83e2549ce086f0e99703a33a80b40e0c94870988851a
2018-10-21 23:54:34.712 [DBG] BTCN: Writing filter headers up to height=20000, hash=0000000008ca11392fa91c4786e59823a002f4868bdb0c1385b12a2844cbc11f
2018-10-21 23:54:34.714 [DBG] BTCN: Sending getcfheaders to 159.203.125.125:18333 (outbound)
2018-10-21 23:54:36.768 [DBG] BTCN: Received headers (num 2000) from 159.203.125.125:18333 (outbound)
2018-10-21 23:54:36.781 [DBG] BTCN: Difficulty retarget at block height 52416
2018-10-21 23:54:36.781 [DBG] BTCN: Old target 1c0f872b (000000000f872b00000000000000000000000000000000000000000000000000)
2018-10-21 23:54:36.781 [DBG] BTCN: New target 1c03f65b (0000000003f65b00000000000000000000000000000000000000000000000000)
2018-10-21 23:54:36.781 [DBG] BTCN: Actual timespan 85h44m18s, adjusted timespan 85h44m18s, target timespan 336h0m0s
2018-10-21 23:54:36.809 [DBG] BTCN: Sending getheaders (locator 0000000003c56a48ca5e57ac893faefe79df193a8fc7da19a8f3220db806e63c, stop 00000000009e2958c15ff9290d571bf9459e93b19765c6801ddeccadbb160a1e) to 159.203.125.125:18333 (outbound)
2018-10-21 23:54:37.358 [DBG] BTCN: Received cfheaders from 159.203.125.125:18333 (outbound)
2018-10-21 23:54:37.359 [DBG] BTCN: Got cfheaders from height=20001 to height=21000, prev_hash=1cf49df621628d958c3e2cc3c2be10d0f97c8081f36efcf05a0102b9cce62ada
2018-10-21 23:54:37.361 [DBG] BTCN: Writing filter headers up to height=21000, hash=0000000000cc441542df3f00075dc5154e2ad72b00fe80e0f25bab8967810447
2018-10-21 23:54:37.362 [DBG] BTCN: Sending getcfheaders to 159.203.125.125:18333 (outbound)
2018-10-21 23:54:37.438 [INF] LNWL: Caught up to height 20000
2018-10-21 23:54:39.257 [DBG] BTCN: Received headers (num 2000) from 159.203.125.125:18333 (outbound)
2018-10-21 23:54:39.258 [INF] BTCN: Processed 8000 blocks in the last 10s (height 52547, 2013-02-15 02:05:32 +0200 IST)
2018-10-21 23:54:39.270 [DBG] BTCN: Difficulty retarget at block height 54432
2018-10-21 23:54:39.270 [DBG] BTCN: Old target 1c03f65b (0000000003f65b00000000000000000000000000000000000000000000000000)
2018-10-21 23:54:39.270 [DBG] BTCN: New target 1c021d8f (00000000021d8f00000000000000000000000000000000000000000000000000)
2018-10-21 23:54:39.271 [DBG] BTCN: Actual timespan 179h23m20s, adjusted timespan 179h23m20s, target timespan 336h0m0s
2018-10-21 23:54:39.302 [DBG] BTCN: Sending getheaders (locator 00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7, stop 00000000009e2958c15ff9290d571bf9459e93b19765c6801ddeccadbb160a1e) to 159.203.125.125:18333 (outbound)
2018-10-21 23:54:39.546 [DBG] BTCN: Received cfheaders from 159.203.125.125:18333 (outbound)
2018-10-21 23:54:39.548 [DBG] BTCN: Got cfheaders from height=21001 to height=22000, prev_hash=477bc213ff0ab1aaa2ed6aa42d0dd9f112b63bbd872eca691de550c5d9472202
2018-10-21 23:54:39.551 [DBG] BTCN: Writing filter headers up to height=22000, hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12
2018-10-21 23:54:39.553 [DBG] BTCN: Sending getcfheaders to 159.203.125.125:18333 (outbound)
^C2018-10-21 23:54:42.060 [INF] LTND: Received SIGINT (Ctrl+C).
2018-10-21 23:54:42.060 [INF] LTND: Shutting down...
2018-10-21 23:54:42.060 [INF] LTND: Gracefully shutting down.
2018-10-21 23:54:42.313 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.313 [INF] BTCN: Lost peer 159.203.125.125:18333 (outbound)
2018-10-21 23:54:42.313 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.313 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.313 [WRN] BTCN: No sync peer candidates available
2018-10-21 23:54:42.319 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.319 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.328 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.328 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.328 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.328 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.329 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.329 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.329 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.329 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.329 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.329 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.329 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.329 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.329 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.329 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.330 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.330 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.330 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.330 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.330 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.330 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.330 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.330 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.330 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.330 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.331 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.331 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.331 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.331 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.331 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.331 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.331 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.331 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.331 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.331 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.331 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.331 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.332 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.332 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.332 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.332 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.332 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.332 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.332 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.332 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.332 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.332 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.332 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.332 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.332 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.332 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.333 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.333 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.333 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.333 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.333 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.333 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.333 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.333 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.333 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.333 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.333 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.333 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.333 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.334 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.334 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.334 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.334 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.334 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.334 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.334 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.334 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.334 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.334 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.334 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.334 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.334 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.334 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.334 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.335 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.335 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.335 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.335 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.335 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.335 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.335 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.335 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.335 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.335 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.335 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.335 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.335 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.335 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.335 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.335 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.336 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.336 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.336 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.336 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.336 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.336 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.336 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.336 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.336 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.336 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.336 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.336 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.336 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.336 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.337 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.337 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.337 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.337 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.337 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.337 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.337 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.337 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.337 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.337 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.337 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.337 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.337 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.337 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.337 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.337 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.337 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.337 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.338 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.338 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.338 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.338 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.338 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.338 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.338 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.338 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.338 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.338 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.338 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.338 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.338 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.338 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.338 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.338 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.338 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.338 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.339 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.339 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.339 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.339 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.339 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.339 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.339 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.339 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.339 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.339 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.339 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.339 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.339 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.339 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.339 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.339 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.339 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.339 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.340 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.340 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.340 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.340 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.340 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.340 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.340 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.340 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.340 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.340 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.340 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.340 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.340 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.340 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.340 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.340 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.340 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.340 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.341 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.341 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.341 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.341 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.341 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.341 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.341 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.341 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.341 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.341 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.341 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.341 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.341 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.341 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.341 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.341 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.341 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.341 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.341 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.342 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.342 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.342 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.342 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.342 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.342 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.342 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.342 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.342 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.342 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.342 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.342 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.342 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.342 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.342 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.342 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.342 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.342 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.342 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.343 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.343 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.343 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.343 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.343 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.343 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.343 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.343 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.343 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.343 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.343 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.343 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.343 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.343 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.343 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.343 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.343 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.343 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.344 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.344 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.344 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.344 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.344 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.344 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.344 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.344 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.344 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.344 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.344 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.344 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.344 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.344 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.344 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.344 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.344 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.344 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.344 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.345 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.345 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.345 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.345 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.345 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.345 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.345 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.345 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.345 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.345 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.345 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.345 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.345 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.345 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.345 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.345 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.345 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.345 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.345 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.346 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.346 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.346 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.346 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.346 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.346 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.346 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.346 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.346 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.346 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.346 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.346 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.346 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.346 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.346 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.346 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.346 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.346 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.346 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.347 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.347 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.347 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.347 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.347 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.347 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.347 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.347 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.347 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.347 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.347 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.347 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.347 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.347 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.347 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.348 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)
2018-10-21 23:54:42.348 [INF] BTCN: Starting cfheaders sync for filter_type=0
2018-10-21 23:54:42.348 [INF] BTCN: Fetching set of checkpointed cfheaders filters from height=22000, hash=88d493aef31b23e4fbcef552ae36cfded9912c71a8acf0d6ab05285a0c381758
2018-10-21 23:54:42.348 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=22
2018-10-21 23:54:42.348 [INF] BTCN: Attempting to query for 32 cfheader batches
2018-10-21 23:54:42.348 [INF] BTCN: Waiting for more block headers, then will start cfheaders sync from height 22000...
2018-10-21 23:54:42.348 [INF] BTCN: Starting cfheaders sync from (block_height=22000, block_hash=0000000021a6d163d8b57e70cdb3db193c10fb1f9f30700c3f14bd23ce915b12) to (block_height=54546, block_hash=00000000aaf40661e269aed9ee007daf9552ef3d8544a3c62286be62ad1313e7)

Test flake: timeout while waiting for ChainService to sync

initial sync, testnet, panic: attempt to write cfheaders out of order

lnd version: Version 0.5.0-beta commit=a3edcf9cd537ea37a502efc91d262e7017a99e6b
btcd version: 2a560b2036bee5e3679ec2133eb6520b2f195213

2018-10-08 11:06:19.826 [INF] BTCN: Starting to query for cfheaders from checkpoint_interval=0
2018-10-08 11:06:19.839 [INF] BTCN: Attempting to query for 1432 cfheader batches
2018-10-08 11:06:19.839 [DBG] BTCN: Sending getcfheaders to 207.154.224.115:18333 (outbound)
2018-10-08 11:06:22.841 [DBG] BTCN: Sending getcfheaders to 207.154.224.115:18333 (outbound)
2018-10-08 11:06:23.008 [DBG] BTCN: Received cfheaders from 207.154.224.115:18333 (outbound)
2018-10-08 11:06:23.010 [DBG] BTCN: Got cfheaders from height=1 to height=1001
2018-10-08 11:06:23.014 [DBG] BTCN: Writing filter headers up to height=1000
2018-10-08 11:06:23.019 [INF] BTCN: Verified 1 filter header in the last 4m52.34s (height 1, 2011-02-03 02:22:08 +0300 MSK)
2018-10-08 11:06:23.019 [DBG] BTCN: Sending getcfheaders to 207.154.224.115:18333 (outbound)
2018-10-08 11:06:23.056 [DBG] BTCN: Received cfheaders from 207.154.224.115:18333 (outbound)
2018-10-08 11:06:25.352 [DBG] BTCN: Received cfheaders from 207.154.224.115:18333 (outbound)
2018-10-08 11:06:25.354 [DBG] BTCN: Got cfheaders from height=1001 to height=2001
panic: couldn't write cfheaders msg: attempt to write cfheaders out of order

goroutine 54 [running]:
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*blockManager).getCheckpointedCFHeaders.func1(0xc4200e0960, 0x1dbeea0, 0xc42048e3f0, 0x1dbec60, 0xc43102a900, 0x501)
	/Users/andrey/GoProjects/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blockmanager.go:871 +0xb2d
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).queryBatch(0xc4204a8000, 0xc431440000, 0x598, 0x598, 0xc430f0be30, 0xc420856000, 0x0, 0x0, 0x0)
	/Users/andrey/GoProjects/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/query.go:422 +0xa85
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*blockManager).getCheckpointedCFHeaders(0xc4202ee160, 0xc431430000, 0x598, 0x598, 0xc4202a2088, 0x500)
	/Users/andrey/GoProjects/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blockmanager.go:774 +0x8d7
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*blockManager).cfHandler.func2(0xc42046a030, 0xc4202ee160, 0x15dc6b, 0xc4308c4840, 0x0, 0x1d146f8)
	/Users/andrey/GoProjects/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blockmanager.go:500 +0x3c5
created by github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*blockManager).cfHandler
	/Users/andrey/GoProjects/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blockmanager.go:440 +0x407

full log:
lnd.log

Deadlock issue with tests

Travis test failed with this goroutine dump:

=== RUN   TestSetup
SIGQUIT: quit
PC=0x48a94d m=0 sigcode=0

goroutine 5 [running]:
racecall()
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/race_amd64.s:380 +0x1d fp=0x7fff4bbf5ba0 sp=0x7fff4bbf5b98 pc=0x48a94d
created by testing.(*T).Run
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/testing/testing.go:789 +0x569

goroutine 1 [chan receive, 9 minutes]:
testing.(*T).Run(0xc4203fe0f0, 0xa22e5a, 0x9, 0xb621f0, 0xc42005bc50)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/testing/testing.go:790 +0x59b
testing.runTests.func1(0xc4203fe0f0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/testing/testing.go:1004 +0xa8
testing.tRunner(0xc4203fe0f0, 0xc42005bd90)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/testing/testing.go:746 +0x16d
testing.runTests(0xc4201145c0, 0xdde120, 0x1, 0x1, 0x30)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/testing/testing.go:1002 +0x522
testing.(*M).Run(0xc42005bf20, 0x29f1ea8)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/testing/testing.go:921 +0x207
main.main()
	github.com/lightninglabs/neutrino/_test/_testmain.go:46 +0x1d4

goroutine 21 [IO wait, 5 minutes]:
internal/poll.runtime_pollWait(0x7f77da173a60, 0x72, 0xdac0c0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc42028a718, 0x72, 0xdac000, 0xda8628, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xfb
internal/poll.(*pollDesc).waitRead(0xc42028a718, 0xc420560000, 0x8000, 0x8000)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc42028a700, 0xc420560000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_unix.go:125 +0x23d
net.(*netFD).Read(0xc42028a700, 0xc420560000, 0x8000, 0x8000, 0x0, 0x0, 0x67)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc42000e090, 0xc420560000, 0x8000, 0x8000, 0x102, 0x10, 0x4259b7)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/net.go:176 +0x88
net.(*TCPConn).Read(0xc42000e090, 0xc420560000, 0x8000, 0x8000, 0xc4201ab701, 0xc4201ab738, 0x7bbf46)
	<autogenerated>:1 +0x6a
crypto/tls.(*block).readFromUntil(0xc420073ec0, 0x7f77da1b7a78, 0xc42000e090, 0x5, 0xc42000e090, 0xc4204ffbc0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:488 +0x11d
crypto/tls.(*Conn).readRecord(0xc420404e00, 0xb62c17, 0xc420404f20, 0x46f971)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:590 +0x142
crypto/tls.(*Conn).Read(0xc420404e00, 0xc420504000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:1134 +0x170
bufio.(*Reader).Read(0xc4200683c0, 0xc42029e080, 0x2, 0x8, 0xc4201abc58, 0x439018, 0x10)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/bufio/bufio.go:213 +0x7b8
github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket.(*Conn).readFull(0xc4203fe2d0, 0xc42029e080, 0x2, 0x8, 0xc42002c000, 0x2002c000)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket/conn.go:542 +0xa0
github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket.(*Conn).advanceFrame(0xc4203fe2d0, 0x40d4360000000000, 0xc420604600, 0x50d8)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket/conn.go:566 +0xe9
github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket.(*Conn).NextReader(0xc4203fe2d0, 0xc42070e130, 0xc4201700c0, 0xc4206d4090, 0x600, 0xc420519120)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket/conn.go:706 +0xdf
github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket.(*Conn).ReadMessage(0xc4203fe2d0, 0xc4205025a0, 0x0, 0x600, 0x600, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket/conn.go:769 +0x40
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).wsInHandler(0xc420416780)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:410 +0xc1
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:1038 +0x20b

goroutine 22 [select, 5 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).wsOutHandler(0xc420416780)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:446 +0x157
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:1039 +0x22d

goroutine 23 [chan receive, 5 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest.(*memWallet).chainSyncer(0xc420400240)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest/memwallet.go:229 +0x77
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest.(*memWallet).Start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest/memwallet.go:161 +0x4d

goroutine 3118 [chan receive, 9 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).WaitForDisconnect(...)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2095
github.com/lightninglabs/neutrino.(*ChainService).peerDoneHandler(0xc42041a000, 0xc42045a2c0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/neutrino.go:1220 +0x85
created by github.com/lightninglabs/neutrino.(*ChainService).outboundPeerConnected
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/neutrino.go:1213 +0x42c

goroutine 5930 [select, 5 minutes]:
github.com/lightninglabs/neutrino.(*blockSubscription).subscriptionHandler(0xc42056ddd0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/blocksubscriptions.go:174 +0x30a
created by github.com/lightninglabs/neutrino.(*ChainService).subscribeBlockMsg
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/blocksubscriptions.go:86 +0x214

goroutine 19837 [select]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).queueHandler(0xc4205bc600)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1721 +0x48f
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2081 +0x3c5

goroutine 19836 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f77da1735e0, 0x72, 0xdac0c0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc4204da218, 0x72, 0xdac000, 0xda8628, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xfb
internal/poll.(*pollDesc).waitRead(0xc4204da218, 0xc4208b2100, 0x18, 0x18)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc4204da200, 0xc4208b2100, 0x18, 0x18, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_unix.go:125 +0x23d
net.(*netFD).Read(0xc4204da200, 0xc4208b2100, 0x18, 0x18, 0x0, 0xc42005d80c, 0xc42005d740)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc4205ea000, 0xc4208b2100, 0x18, 0x18, 0x4159b0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/net.go:176 +0x88
net.(*TCPConn).Read(0xc4205ea000, 0xc4208b2100, 0x18, 0x18, 0x10007c42005d7f0, 0x0, 0x18)
	<autogenerated>:1 +0x6a
io.ReadAtLeast(0x7f77da1b7a78, 0xc4205ea000, 0xc4208b2100, 0x18, 0x18, 0x18, 0xc42005d838, 0x439018, 0x20)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/io/io.go:309 +0x94
io.ReadFull(0x7f77da1b7a78, 0xc4205ea000, 0xc4208b2100, 0x18, 0x18, 0x9df180, 0xc4208b20c0, 0x100000000020)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/io/io.go:327 +0x73
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/wire.readMessageHeader(0x7f77da1b7a78, 0xc4205ea000, 0xc42019a2a0, 0x20001117d, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/wire/message.go:204 +0xc8
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/wire.ReadMessageWithEncodingN(0x7f77da1b7a78, 0xc4205ea000, 0x12141c160001117d, 0x7f7700000002, 0xc4205ea000, 0xc42005dc10, 0x477e56, 0xc, 0x0, 0x437164, ...)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/wire/message.go:340 +0x61
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).readMessage(0xc4205bc600, 0x4500000002, 0xdebc00, 0xc4205b40b0, 0xc4205b40b8, 0x8, 0x8, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1146 +0x10d
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).inHandler(0xc4205bc600)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1468 +0x146
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2080 +0x3a0

goroutine 19835 [select]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).stallHandler(0xc4205bc600)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1331 +0x33f
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2079 +0x37b

goroutine 3120 [select]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).stallHandler(0xc4205bc000)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1331 +0x33f
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2079 +0x37b

goroutine 19888 [select]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).queueHandler(0xc4205bc300)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1721 +0x48f
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2081 +0x3c5

goroutine 19886 [select]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).stallHandler(0xc4205bc300)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1331 +0x33f
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2079 +0x37b

goroutine 3046 [select, 5 minutes]:
github.com/lightninglabs/neutrino.(*blockManager).blockHandler(0xc4204d67e0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/blockmanager.go:337 +0x316
created by github.com/lightninglabs/neutrino.(*blockManager).Start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/blockmanager.go:200 +0x12b

goroutine 19890 [select, 1 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).pingHandler(0xc4205bc300)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1929 +0x168
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2083 +0x40f

goroutine 3045 [select, 9 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/addrmgr.(*AddrManager).addressHandler(0xc42044ea00)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/addrmgr/addrmanager.go:342 +0x17d
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/addrmgr.(*AddrManager).Start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/addrmgr/addrmanager.go:551 +0x13f

goroutine 3326 [chan send, 8 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).inHandler(0xc4205bc900)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1505 +0x8ad
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2080 +0x3a0

goroutine 19833 [chan receive, 5 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).WaitForDisconnect(...)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2095
github.com/lightninglabs/neutrino.(*ChainService).peerDoneHandler(0xc42041a000, 0xc42045a420)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/neutrino.go:1220 +0x85
created by github.com/lightninglabs/neutrino.(*ChainService).outboundPeerConnected
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/neutrino.go:1213 +0x42c

goroutine 3140 [select, 1 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).pingHandler(0xc4205bc000)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1929 +0x168
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2083 +0x40f

goroutine 19838 [select, 1 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).outHandler(0xc4205bc600)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1855 +0x19b
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2082 +0x3ea

goroutine 4167 [chan receive, 8 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest.(*memWallet).chainSyncer(0xc4201da240)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest/memwallet.go:229 +0x77
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest.(*memWallet).Start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest/memwallet.go:161 +0x4d

goroutine 3092 [select, 5 minutes]:
github.com/lightninglabs/neutrino.(*ChainService).peerHandler(0xc42041a000)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/neutrino.go:911 +0x5ae
created by github.com/lightninglabs/neutrino.(*ChainService).Start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/neutrino.go:1257 +0xa3

goroutine 531 [chan receive, 5 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest.(*memWallet).chainSyncer(0xc4204003f0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest/memwallet.go:229 +0x77
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest.(*memWallet).Start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest/memwallet.go:161 +0x4d

goroutine 533 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x7f77da1738e0, 0x72, 0xdac0c0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc42028a218, 0x72, 0xdac000, 0xda8628, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xfb
internal/poll.(*pollDesc).waitRead(0xc42028a218, 0xc420530000, 0x8000, 0x8000)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc42028a200, 0xc420530000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_unix.go:125 +0x23d
net.(*netFD).Read(0xc42028a200, 0xc420530000, 0x8000, 0x8000, 0xc4200ff1f8, 0xc4200ff1f0, 0x8)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc420514028, 0xc420530000, 0x8000, 0x8000, 0x102, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/net.go:176 +0x88
net.(*TCPConn).Read(0xc420514028, 0xc420530000, 0x8000, 0x8000, 0x7f77da2be000, 0xc4201a9738, 0x7bbf46)
	<autogenerated>:1 +0x6a
crypto/tls.(*block).readFromUntil(0xc4204de390, 0x7f77da1b7a78, 0xc420514028, 0x5, 0xc420514028, 0x46f810)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:488 +0x11d
crypto/tls.(*Conn).readRecord(0xc42043c000, 0xb62c17, 0xc42043c120, 0x2bb9)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:590 +0x142
crypto/tls.(*Conn).Read(0xc42043c000, 0xc4204e6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:1134 +0x170
bufio.(*Reader).Read(0xc4200687e0, 0xc4200ff210, 0x2, 0x8, 0xc4201a9c58, 0x439018, 0x10)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/bufio/bufio.go:213 +0x7b8
github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket.(*Conn).readFull(0xc4203fe4b0, 0xc4200ff210, 0x2, 0x8, 0xc42002c000, 0x2002c000)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket/conn.go:542 +0xa0
github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket.(*Conn).advanceFrame(0xc4203fe4b0, 0x4010000000000000, 0xc420166000, 0x4)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket/conn.go:566 +0xe9
github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket.(*Conn).NextReader(0xc4203fe4b0, 0xc4205a9790, 0xc420179c20, 0xc420076540, 0xfe00, 0xc4201647e0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket/conn.go:706 +0xdf
github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket.(*Conn).ReadMessage(0xc4203fe4b0, 0xc4205021e0, 0xeb00, 0xfe00, 0xfe00, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket/conn.go:769 +0x40
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).wsInHandler(0xc420416b40)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:410 +0xc1
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:1038 +0x20b

goroutine 481 [IO wait, 5 minutes]:
internal/poll.runtime_pollWait(0x7f77da1739a0, 0x72, 0xdac0c0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc4204da298, 0x72, 0xdac000, 0xda8628, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xfb
internal/poll.(*pollDesc).waitRead(0xc4204da298, 0xc420646000, 0x8000, 0x8000)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc4204da280, 0xc420646000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_unix.go:125 +0x23d
net.(*netFD).Read(0xc4204da280, 0xc420646000, 0x8000, 0x8000, 0x59, 0x46f810, 0x46f810)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc420514010, 0xc420646000, 0x8000, 0x8000, 0x102, 0x59, 0x60)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/net.go:176 +0x88
net.(*TCPConn).Read(0xc420514010, 0xc420646000, 0x8000, 0x8000, 0x0, 0xc420057738, 0x7bbf46)
	<autogenerated>:1 +0x6a
crypto/tls.(*block).readFromUntil(0xc4205d9050, 0x7f77da1b7a78, 0xc420514010, 0x5, 0xc420514010, 0xc4204ffbc0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:488 +0x11d
crypto/tls.(*Conn).readRecord(0xc42003b180, 0xb62c17, 0xc42003b2a0, 0x46f971)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:590 +0x142
crypto/tls.(*Conn).Read(0xc42003b180, 0xc420505000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:1134 +0x170
bufio.(*Reader).Read(0xc4200685a0, 0xc420270728, 0x2, 0x8, 0xc420057c58, 0x439018, 0x8)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/bufio/bufio.go:213 +0x7b8
github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket.(*Conn).readFull(0xc4203fe3c0, 0xc420270728, 0x2, 0x8, 0xc42002d300, 0x2002d300)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket/conn.go:542 +0xa0
github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket.(*Conn).advanceFrame(0xc4203fe3c0, 0x409d940000000000, 0xc420492000, 0x765)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket/conn.go:566 +0xe9
github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket.(*Conn).NextReader(0xc4203fe3c0, 0xc42067da40, 0xc42051f320, 0xc420146180, 0x600, 0xc4208678c0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket/conn.go:706 +0xdf
github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket.(*Conn).ReadMessage(0xc4203fe3c0, 0xc420503da0, 0x0, 0x600, 0x600, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/btcsuite/websocket/conn.go:769 +0x40
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).wsInHandler(0xc420416a00)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:410 +0xc1
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:1038 +0x20b

goroutine 534 [select, 9 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).wsOutHandler(0xc420416b40)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:446 +0x157
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:1039 +0x22d

goroutine 1385 [chan receive, 9 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest.(*memWallet).chainSyncer(0xc420400750)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest/memwallet.go:229 +0x77
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest.(*memWallet).Start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest/memwallet.go:161 +0x4d

goroutine 535 [chan receive, 9 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest.(*memWallet).chainSyncer(0xc4204005a0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest/memwallet.go:229 +0x77
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest.(*memWallet).Start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/integration/rpctest/memwallet.go:161 +0x4d

goroutine 530 [select, 5 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).wsOutHandler(0xc420416a00)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:446 +0x157
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:1039 +0x22d

goroutine 19884 [chan receive, 5 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).WaitForDisconnect(...)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2095
github.com/lightninglabs/neutrino.(*ChainService).peerDoneHandler(0xc42041a000, 0xc4200824d0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/neutrino.go:1220 +0x85
created by github.com/lightninglabs/neutrino.(*ChainService).outboundPeerConnected
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/neutrino.go:1213 +0x42c

goroutine 19889 [select, 1 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).outHandler(0xc4205bc300)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1855 +0x19b
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2082 +0x3ea

goroutine 3048 [select, 5 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/connmgr.(*ConnManager).connHandler(0xc420401170)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/connmgr/connmanager.go:222 +0x208
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/connmgr.(*ConnManager).Start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/connmgr/connmanager.go:365 +0x130

goroutine 3139 [select, 1 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).outHandler(0xc4205bc000)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1855 +0x19b
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2082 +0x3ea

goroutine 19887 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f77da1736a0, 0x72, 0xdac0c0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc42028a198, 0x72, 0xdac000, 0xda8628, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xfb
internal/poll.(*pollDesc).waitRead(0xc42028a198, 0xc42072e000, 0x18, 0x18)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc42028a180, 0xc42072e0e0, 0x18, 0x18, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_unix.go:125 +0x23d
net.(*netFD).Read(0xc42028a180, 0xc42072e0e0, 0x18, 0x18, 0x0, 0xc4207e180c, 0xc4207e1740)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc42000e008, 0xc42072e0e0, 0x18, 0x18, 0x4159b0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/net.go:176 +0x88
net.(*TCPConn).Read(0xc42000e008, 0xc42072e0e0, 0x18, 0x18, 0x10007c4207e17f0, 0x0, 0x18)
	<autogenerated>:1 +0x6a
io.ReadAtLeast(0x7f77da1b7a78, 0xc42000e008, 0xc42072e0e0, 0x18, 0x18, 0x18, 0xc4207e1838, 0x439018, 0x20)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/io/io.go:309 +0x94
io.ReadFull(0x7f77da1b7a78, 0xc42000e008, 0xc42072e0e0, 0x18, 0x18, 0x9df180, 0xc42072e0c0, 0x100000000020)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/io/io.go:327 +0x73
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/wire.readMessageHeader(0x7f77da1b7a78, 0xc42000e008, 0xc42019a2a0, 0x20001117d, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/wire/message.go:204 +0xc8
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/wire.ReadMessageWithEncodingN(0x7f77da1b7a78, 0xc42000e008, 0x12141c160001117d, 0x7f7700000002, 0xc42000e008, 0xc4207e1c10, 0x477e56, 0xc, 0x0, 0x437164, ...)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/wire/message.go:340 +0x61
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).readMessage(0xc4205bc300, 0x4500000002, 0xdebc00, 0xc42029e0d0, 0xc42029e0d8, 0x8, 0x8, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1146 +0x10d
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).inHandler(0xc4205bc300)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1468 +0x146
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2080 +0x3a0

goroutine 3121 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f77da173820, 0x72, 0xdac0c0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc4205b6098, 0x72, 0xdac000, 0xda8628, 0xffffffffffffffff)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xfb
internal/poll.(*pollDesc).waitRead(0xc4205b6098, 0xc420814000, 0x18, 0x18)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc4205b6080, 0xc4208140e0, 0x18, 0x18, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_unix.go:125 +0x23d
net.(*netFD).Read(0xc4205b6080, 0xc4208140e0, 0x18, 0x18, 0x0, 0xc4207db80c, 0xc4207db740)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc420514068, 0xc4208140e0, 0x18, 0x18, 0x4159b0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/net.go:176 +0x88
net.(*TCPConn).Read(0xc420514068, 0xc4208140e0, 0x18, 0x18, 0x10007c4207db7f0, 0x0, 0x18)
	<autogenerated>:1 +0x6a
io.ReadAtLeast(0x7f77da1b7a78, 0xc420514068, 0xc4208140e0, 0x18, 0x18, 0x18, 0xc4207db838, 0x439018, 0x20)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/io/io.go:309 +0x94
io.ReadFull(0x7f77da1b7a78, 0xc420514068, 0xc4208140e0, 0x18, 0x18, 0x9df180, 0xc4208140c0, 0x100000000020)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/io/io.go:327 +0x73
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/wire.readMessageHeader(0x7f77da1b7a78, 0xc420514068, 0xc42019a2a0, 0x20001117d, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/wire/message.go:204 +0xc8
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/wire.ReadMessageWithEncodingN(0x7f77da1b7a78, 0xc420514068, 0x12141c160001117d, 0x7f7700000002, 0xc420514068, 0xc4207dbc10, 0x477e56, 0xc, 0x0, 0x437164, ...)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/wire/message.go:340 +0x61
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).readMessage(0xc4205bc000, 0x4500000002, 0xdebc00, 0xc4208120b0, 0xc4208120b8, 0x8, 0x8, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1146 +0x10d
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).inHandler(0xc4205bc000)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1468 +0x146
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2080 +0x3a0

goroutine 3165 [select, 5 minutes]:
github.com/lightninglabs/neutrino.(*ChainService).Rescan(0xc42041a000, 0xc4204c1280, 0x5, 0x8, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/rescan.go:300 +0x1c2b
github.com/lightninglabs/neutrino.(*Rescan).Start.func1(0xc4202c7a40, 0xc4202c7aa0)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/rescan.go:742 +0x152
created by github.com/lightninglabs/neutrino.(*Rescan).Start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/rescan.go:740 +0x9b

goroutine 3138 [select]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).queueHandler(0xc4205bc000)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1721 +0x48f
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2081 +0x3c5

goroutine 19839 [select, 1 minutes]:
github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).pingHandler(0xc4205bc600)
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:1929 +0x168
created by github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightninglabs/neutrino/vendor/github.com/roasbeef/btcd/peer/peer.go:2083 +0x40f

rax    0x417f30
rbx    0xc4206c1030
rcx    0xc4206b0558
rdx    0x8fc41e
rdi    0x2a6f428
rsi    0xc4206b0558
rbp    0xc4206b0470
rsp    0x7fff4bbf5b98
r8     0xc
r9     0xc4206b0180
r10    0xdec320
r11    0x4000000000000000
r12    0xc4206b03d0
r13    0xdec820
r14    0xc420448300
r15    0xc4206b02a0
rip    0x48a94d
rflags 0x283
cs     0x33
fs     0x0
gs     0x0
*** Test killed with quit: ran too long (10m0s).
FAIL	github.com/lightninglabs/neutrino	600.014s

Stall downloading cfheaders when peers misbehave

I've noticed this on bch fork of this codebase. Part of the problem is #130 but simply disconnecting and banning the peer like is done in the PR causes the download to stall.

Another separate issue seems to be either peers timing out or being very slow to respond to the message. In the later case I think the timeout fires and the goroutine continues to the next iteration of the loop, but it appears the message can still come in after a timeout putting everything in a bad state and causing a stall.

Thus far I've been running with these patches and I've synced a number of times so far with connections to known bad nodes and it's made it to the tip without stalling.
https://github.com/gcash/neutrino/pull/20/files

Fix issue where stopping the sync can break it and prevent correct restart

Currently, the synchronization process can be interrupted while a cfheaders message is being processed, leaving the cfheaders not caught up to the block headers. Restarting the synchronization does not catch up the cfheaders to the block headers as the code assumes they're already caught up; this breaks synchronization.

blockmanager: aggressively pipeline getheaders requests

Currently when fetching block headers, we wait until we finish processing the current batch before we fetch the very next batch. This is inefficient, as it doesn't allow us to keep pipeline and keep the network saturated while we're fetching and verifying headers. Instead, once we receive a batch of headers, we should fetch the very next immediately. It may even be worth breaking fetching and processing out into distinct goroutines. If this is done, then it'll be possibly to nearly fetch the entire set of headers, while we're busy with I/O and header verification.

While we're at it, we should continue always fetch headers beyond the last checkpoint. Much of this sync code was borrowed from btcd. In our case however, we don't care about fully validation, only PoW so we can more aggressively pipeline our block fetching, and stop using getblocks all together.

Welcome Screen pagination UI is overlapping text

Test flake: Timed out after 30s waiting for cfheaders synchronization.

The following error is seen regularly during sync tests:

--- FAIL: TestNeutrinoSync (20.21s)
    --- FAIL: TestNeutrinoSync/initial_sync (31.05s)
        sync_test.go:288: Couldn't sync ChainService: Timed out after 30s waiting for cfheaders synchronization.

The reason seems to be the undefined order at which we query our peers, where each query is sent to only one peer at a time:

neutrino/query.go

Line 247 in 9a42f7d

peerGoroutine := func(sp *ServerPeer, quit <-chan struct{},

During the test in question, we are connected to 1 "good" and two "bad" peers. So if we are unlucky to not query the good peer at least once during the timeout, the test will fail.

We should be smarter about penalizing peers that are not answering our queries, to converge on the good peers.

NewChainService does not traverse through all supplied Neutrino Peers before returning with error

With multiple neutrino peers supplied through lnd.conf/neutrino.Config structure, it seems that NewChainService() will try to resolve the peers' addresses and try to connect to them. However as soon as one of these addresses are deemed invalid, NewChainService() stops attempting and simply returns with error immediately, without trying the remaining peer addresses.

At first glance it can be a fairly trivial change to remedy this? But not sure if there are other implications, like whether the neutrino.Config structure should be updated to remove the invalid addresses, etc.

        // Start up persistent peers.
	permanentPeers := cfg.ConnectPeers
	if len(permanentPeers) == 0 {
		permanentPeers = cfg.AddPeers
	}

	var peerConnErr error
	var peersConnected int

	for _, addr := range permanentPeers {
		tcpAddr, err := s.addrStringToNetAddr(addr)

		if err != nil {
			peerConnErr = err
		} else {
			peersConnected++
			go s.connManager.Connect(&connmgr.ConnReq{
				Addr:      tcpAddr,
				Permanent: true,
			})
		}
	}

	if peersConnected != 0 {
		peerConnErr = nil
	}

	return &s, peerConnErr

I can create a Pull Request if this seems like that right direction to go?

neutrino: implement BIP 151

Currently neutrino is used primarily within lnd as an optional chain backend. Within the current LN protocol, all traffic is encrypted by default. However, our Bitcoin p2p traffic would still currently be in plaintext and unauthenticated. In order to make our traffic a bit more uniform, we should implement BIP 151 and switch to it for supported nodes. This is also important as before the filters are committed, our current security model relies heavily on a non-partitioning assumption, and several ISP level attacks can be mitigated with added encryption.

I'm tracking the issue here, but a bulk of the code will likely end up living in the peer package of btcsuite/btcd.

neutrino: never write filters to disk unless signaled, instead utilize LRU cache

Currently we always write every filter we need to fetch to disk. This results in high-ish disk and I/O utilization when booting back up after a long dormant period, or aggressive usage of the GetUTXO call. For "normal" wallets, they'll almost never need to re-match a filter once it's been fetched unless they're doing something like a key import. As a result, we should only actually keep a limited (and configurable) set of filters in memory.

Steps To Completion

• Modify the GetCfilter method to only write to disk if specified by the caller. If we don't write the filter to disk, then we should write it to the cache.

• Create a new generic (as in []byte) LRU cache. This can double for usage to cache blocks, as well as filters. It may be the case that a more advanced eviction strategy will be more optimal, but we'll at least start with this just to pick the low hanging fruit.

• Modify the GetCfilter method to first check the cache, then disk if a filter is requested.

neutrino: implement parallel header download via header checkpoints

Currently, we sync the chain purely serially from a single peers. However, due to the existence of header checkpoints, it's actually possible for us to partition up the header space between the checkpoints and download them all in parallel from several distinct peers. This has the potential to speed up initial sync a good bit as we can start to cut off slow peers, and re-assign ranges to responsive peers.

Note that we'll likely want to address #66 before this, as that should refactor the code significantly and also switch over to pure usage of getheaders.

Clarify roasbeef dependencies

Currently the glide.yaml file has the btcsuite dependencies replaces with the forks on roasbeefs account. That's fair enough.

I would however suggest the following:

• Specify the version as a branch instead of a commit hash. This makes it easier to follow the changes that will end up here.
• Keep a list in this repo with the features that btcsuite is lacking in order to switch back to btcsuite.

Error on building the latest master without GO111MODULE=on

Runnning go build it returns

# github.com/lightninglabs/neutrino
neutrino/neutrino.go:1206:4: cannot use sp.OnVersion (type func(*peer.Peer, *wire.MsgVersion) *wire.MsgReject) as type func(*peer.Peer, *wire.MsgVersion) in field value
neutrino/rescan.go:982:22: undefined: txscript.ComputePkScript

neutrino: add support for side-header loading on initial sync

One way to speed up the initial sync for neutrino is to actually package a set of headers (both regular and filter headers) along side the application that packages neutrino. This would allow one to package a set of (possibly compressed) headers that will be written to disk on start up before we start to fetch headers from the network.

Steps To Completion

• Modify the initial constructor to add a new set of functional options for side loaded headers.

• On start up, before syncing, if this is IBD, we should read these headers and write them directly to disk. In the suggested model, we skip verification all together, as it's assumed that these headers are being fetched from a trusted source.

• As a bonus, we can also compress the set of headers, and decompress them within neutrino. This may be useful for contexts such as mobile applications, where reducing the size of the apk is desirable. An example of a header specific compression scheme we may want to look at is: https://github.com/petertodd/rust-bitcoin-headers#how-it-works. There likely some additional optimizations on top of this that we can explore and later implement.

blockmanager: once we're at tip, start to use sendheaders to learn of new chains

Currently we rely on inv's at the chain tip to learn of new blocks. However, with BIP 130 nodes can opt in to instead receive a headers message once the main chain is being extended. We can take advantage of this to eliminate the extra set of round trips involved to fetch the new set of headers. Instead, we can learn of the new tip almost instantly.

[Questions

rescan: only fetch blocks w/ witness data if we matched on an outpoint

One benefit of the switch to segwit, is that light clients are able to fetch blocks without witness data if they don't actually need access to the witness items. However, we don't currently take advantage of the optimization. As a result, our bandwidth usage is higher than it actually should be.

In lnd (the biggest user of this project), we only need access to the witness data of a block if we're attempting to extract the pre-image from the witness of an htlc success sweeping transaction. In all other cases, we can do just fine with only the witness data. We should expose this information to callers of the rescan API to allow them to signal if they'd like the block w/ full witness data if the outpoint in question matters. Note that in the near future, we'll switch to matching on the previous output scripts instead.

Steps To Completion

• Modify the rescan logic to store previous output scripts and output scripts in distinct slice .

• When matching a block, only fetch the block w/ witness data if a previous output script matched. Note that this means we should always first match the previous output scripts to ensure that we don't miss any witness data.

• If the filter does match an input, then we should fetch the block w/ witness data. Otherwise, just fetch the regular block.

• Similarly, the GetUtxo method should be modified to add an additional functional parameter to allow callers to signal if they would like the spending transaction (if it exists) with witness data or not.

Implement a batched version of GetUtxo

An application that requires many concurrent calls to GetUtxo will find that it requests the same blocks from the network multiple times. A batched version of GetUtxo could solve this and would also reduce the amount of IO by not repeatedly fetching the same filters.

Implementation plan:
Create a filter consisting of all outpoints in the query.
Figure out which blocks might match by requesting all relevant filters.
Grab those blocks from the network and check the transactions.

Add sendheaders support and improve reorg handling

After issue #2 is resolved, it should be easier to add sendheaders support. In addition, reorgs right now can't be more than 2000 blocks or go past the most recent restart due to how they're handled; sendheaders support should include a fix for this.

Call to `NewAddress` while filters are fetchin causes deadlock

Currently in the Lightning App, we'll query for a new address at startup, both on initial startup and on catchup after the app has been offline. This causes a deadlock

1 @ 0x102ea0a 0x102eabe 0x10059db 0x1005775 0x14c011e 0x14d7aff 0x150425d 0x150f20e 0x1345d60 0x1503e64 0x176b527 0x18c475a 0x16fa339 0x18094e6 0x16e6657 0x16a76c8 0x16aa908 0x16b017f 0x105b421
#	0x14c011d	github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*Rescan).Update+0xad				/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:1102
#	0x14d7afe	github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/chain.(*NeutrinoClient).NotifyReceived+0x48e	/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/chain/neutrino.go:428
#	0x150425c	github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet.(*Wallet).newAddress+0x3bc			/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet/wallet.go:2801
#	0x150f20d	github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet.(*Wallet).NewAddress.func1+0xbd		/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet/wallet.go:2772
#	0x1345d5f	github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/walletdb.Update+0x1cf				/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/walletdb/interface.go:229
#	0x1503e63	github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet.(*Wallet).NewAddress+0x93			/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet/wallet.go:2769
#	0x176b526	github.com/lightningnetwork/lnd/lnwallet/btcwallet.(*BtcWallet).NewAddress+0xb6						/Users/johan/golang/src/github.com/lightningnetwork/lnd/lnwallet/btcwallet/btcwallet.go:255
#	0x18c4759	main.(*rpcServer).NewAddress+0x69											/Users/johan/golang/src/github.com/lightningnetwork/lnd/rpcserver.go:528
#	0x16fa338	github.com/lightningnetwork/lnd/lnrpc._Lightning_NewAddress_Handler.func1+0x88						/Users/johan/golang/src/github.com/lightningnetwork/lnd/lnrpc/rpc.pb.go:6287
#	0x18094e5	github.com/lightningnetwork/lnd/macaroons.(*Service).UnaryServerInterceptor.func1+0x155					/Users/johan/golang/src/github.com/lightningnetwork/lnd/macaroons/service.go:126
#	0x16e6656	github.com/lightningnetwork/lnd/lnrpc._Lightning_NewAddress_Handler+0x166						/Users/johan/golang/src/github.com/lightningnetwork/lnd/lnrpc/rpc.pb.go:6289
#	0x16a76c7	github.com/lightningnetwork/lnd/vendor/google.golang.org/grpc.(*Server).processUnaryRPC+0x9c7				/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/google.golang.org/grpc/server.go:781
#	0x16aa907	github.com/lightningnetwork/lnd/vendor/google.golang.org/grpc.(*Server).handleStream+0x1317				/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/google.golang.org/grpc/server.go:981
#	0x16b017e	github.com/lightningnetwork/lnd/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1+0x9e			/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/google.golang.org/grpc/server.go:551

1 @ 0x102ea0a 0x102eabe 0x103f2b4 0x103efcd 0x1063048 0x124775b 0x1247448 0x1768798 0x1768c70 0x1345d23 0x1506975 0x14ee765 0x105b421
#	0x103efcc	sync.runtime_SemacquireMutex+0x3c												/usr/local/Cellar/go/1.10.3/libexec/src/runtime/sema.go:71
#	0x1063047	sync.(*Mutex).Lock+0x107													/usr/local/Cellar/go/1.10.3/libexec/src/sync/mutex.go:134
#	0x124775a	github.com/lightningnetwork/lnd/vendor/github.com/coreos/bbolt.(*DB).beginRWTx+0x5a						/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/github.com/coreos/bbolt/db.go:567
#	0x1247447	github.com/lightningnetwork/lnd/vendor/github.com/coreos/bbolt.(*DB).Begin+0x37							/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/github.com/coreos/bbolt/db.go:516
#	0x1768797	github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/walletdb/bdb.(*db).beginTx+0x37				/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/walletdb/bdb/db.go:281
#	0x1768c6f	github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/walletdb/bdb.(*db).BeginReadWriteTx+0x2f			/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/walletdb/bdb/db.go:293
#	0x1345d22	github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/walletdb.Update+0x192					/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/walletdb/interface.go:225
#	0x1506974	github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet.(*Wallet).handleChainNotifications.func2+0x224	/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet/chainntfns.go:56
#	0x14ee764	github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet.(*Wallet).handleChainNotifications+0xcf4		/Users/johan/golang/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/btcwallet/wallet/chainntfns.go:144

I've been able to reliably reproduce by syncing the app, exiting, then restarting it after a few block has been mined on testnet.

The reason this is happening is that a call to NewAddress will open a database RW transaction: https://github.com/btcsuite/btcwallet/blob/7b84dc25a61634c450d21ec7f47d5e916eb88fdb/wallet/wallet.go#L2768

At the same time, the ongoing rescan needs to obtain database RW access to update the current best height: https://github.com/btcsuite/btcwallet/blob/7b84dc2/wallet/chainntfns.go#L52

The call to NewAddress will call the chain client's NotifyReceived with the new address: https://github.com/btcsuite/btcwallet/blob/7b84dc25a61634c450d21ec7f47d5e916eb88fdb/wallet/wallet.go#L2800

Which will attempt to notify the ongoing rescan about the update: https://github.com/btcsuite/btcwallet/blob/7b84dc2/chain/neutrino.go#L428

This call will block trying the send on the updateChan:

neutrino/rescan.go

Line 1102 in 31332f9

r.updateChan <- uo

Handle conflicting cfheader/cfilter information given by different peers

When issue #2 is resolved, it will be much easier to handle conflicting cfheader and cfilter information received from different peers. To test that this works correctly, the btcd rpctest framework will need an addition to allow p2p man-in-the-middle attacks in test cases.

forAllOutboundPeers & forAllPeers

Was taking a look at the source code and found this:

neutrino/neutrino.go

Lines 106 to 121 in e89b367

	// forAllOutboundPeers is a helper function that runs closure on all outbound
	// peers known to peerState.
	func (ps *peerState) forAllOutboundPeers(closure func(sp *ServerPeer)) {
	for _, e := range ps.outboundPeers {
	closure(e)
	}
	for _, e := range ps.persistentPeers {
	closure(e)
	}
	}
	// forAllPeers is a helper function that runs closure on all peers known to
	// peerState.
	func (ps *peerState) forAllPeers(closure func(sp *ServerPeer)) {
	ps.forAllOutboundPeers(closure)
	}

Shouldn't forAllOutboundPeers only apply closure to ps.outboundPeers?

Deadlock issue with tests

Tests sometimes deadlock with a goroutine dump similar to the below:

SIGQUIT: quit
PC=0x45ec33 m=0 sigcode=0
goroutine 18 [syscall]:
runtime.notetsleepg(0xbb5a40, 0x253ff413f, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/lock_futex.go:227 +0x42 fp=0xc420035f60 sp=0xc420035f30 pc=0x411102
runtime.timerproc()
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/time.go:216 +0x305 fp=0xc420035fe0 sp=0xc420035f60 pc=0x44cb25
runtime.goexit()
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420035fe8 sp=0xc420035fe0 pc=0x45d7b1
created by runtime.addtimerLocked
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/time.go:122 +0xed
goroutine 1 [chan receive, 9 minutes]:
testing.(*T).Run(0xc4203ec0f0, 0x85d27b, 0xe, 0x9996c0, 0x47d526)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4203ec0f0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4203ec0f0, 0xc42005dd78)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/testing/testing.go:746 +0xd0
testing.runTests(0xc4204646c0, 0xba7fd0, 0x1, 0x1, 0x100090100000160)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc42005df18, 0x3)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/testing/testing.go:921 +0x111
main.main()
	github.com/lightningnetwork/lnd/chainntnfs/_test/_testmain.go:94 +0x1b2
goroutine 7 [semacquire, 9 minutes]:
sync.runtime_SemacquireMutex(0xc4204aa17c, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/sema.go:71 +0x3d
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/sync/mutex.go:134 +0xee
github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*DB).Close(0xc4204aa000, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/db.go:392 +0x4f
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/bdb.(*db).Close(0xc4204aa000, 0xc420100170, 0xc4201000f0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/bdb/db.go:310 +0x2f
github.com/lightningnetwork/lnd/chainntnfs_test.TestInterfaces.func1()
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/chainntnfs/interface_test.go:978 +0x4a
github.com/lightningnetwork/lnd/chainntnfs_test.TestInterfaces(0xc4203ec1e0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/chainntnfs/interface_test.go:1018 +0x9df
testing.tRunner(0xc4203ec1e0, 0x9996c0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/testing/testing.go:789 +0x2de
goroutine 9 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x7f20cf8abc80, 0x72, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420212298, 0x72, 0xffffffffffffff00, 0xb77f80, 0xb74548)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420212298, 0xc4200f6000, 0x1000, 0x1000)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420212280, 0xc4200f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc420212280, 0xc4200f6000, 0x1000, 0x1000, 0x0, 0x0, 0xc4201ee4c0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc42020c010, 0xc4200f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/net.go:176 +0x6d
crypto/tls.(*block).readFromUntil(0xc420204570, 0x7f20cf90dee0, 0xc42020c010, 0x5, 0xc42020c010, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:488 +0x95
crypto/tls.(*Conn).readRecord(0xc420214000, 0x99a317, 0xc420214120, 0xc400000008)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:590 +0xe0
crypto/tls.(*Conn).Read(0xc420214000, 0xc4202f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/crypto/tls/conn.go:1134 +0x110
bufio.(*Reader).Read(0xc42006a3c0, 0xc42028c228, 0x2, 0x8, 0xc42005bd90, 0x412668, 0x8)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/bufio/bufio.go:213 +0x30b
github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/websocket.(*Conn).readFull(0xc4203ec2d0, 0xc42028c228, 0x2, 0x8, 0x10, 0xc42004f5c0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/websocket/conn.go:542 +0x73
github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/websocket.(*Conn).advanceFrame(0xc4203ec2d0, 0x200, 0xc4201eb800, 0x1ce)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/websocket/conn.go:566 +0x8a
github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/websocket.(*Conn).NextReader(0xc4203ec2d0, 0xc42004f5a0, 0xc42005bf58, 0x4069ea, 0xc420024540, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/websocket/conn.go:706 +0x62
github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/websocket.(*Conn).ReadMessage(0xc4203ec2d0, 0xc420024540, 0x100, 0x600, 0x600, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/btcsuite/websocket/conn.go:769 +0x2f
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).wsInHandler(0xc42008a8c0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:410 +0x7d
created by github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).start
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:1038 +0x17d
goroutine 10 [select, 9 minutes]:
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).wsOutHandler(0xc42008a8c0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:446 +0x10b
created by github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/rpcclient.(*Client).start
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/rpcclient/infrastructure.go:1039 +0x19f
goroutine 11 [chan receive, 9 minutes]:
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/integration/rpctest.(*memWallet).chainSyncer(0xc4203ee240)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/integration/rpctest/memwallet.go:203 +0x270
created by github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/integration/rpctest.(*memWallet).Start
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/integration/rpctest/memwallet.go:160 +0x3f
goroutine 153 [select, 9 minutes]:
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*blockSubscription).subscriptionHandler.func1(0xc4201dc6c0, 0x1)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blocksubscriptions.go:133 +0x1c6
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*blockSubscription).subscriptionHandler(0xc42019a000)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blocksubscriptions.go:158 +0x131
created by github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).subscribeBlockMsg
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blocksubscriptions.go:86 +0x1af
goroutine 145 [select, 9 minutes]:
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/connmgr.(*ConnManager).connHandler(0xc4203eebd0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/connmgr/connmanager.go:222 +0x1a7
created by github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/connmgr.(*ConnManager).Start
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/connmgr/connmanager.go:365 +0xea
goroutine 143 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc4204aa194)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/sema.go:56 +0x39
sync.(*RWMutex).RLock(0xc4204aa188)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/sync/rwmutex.go:50 +0x49
github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*DB).beginTx(0xc4204aa000, 0xc4200598c8, 0x411f07, 0xc42000e388)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/db.go:475 +0x5d
github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*DB).Begin(0xc4204aa000, 0x753a00, 0x200204aa1a0, 0x8, 0x8)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/db.go:463 +0x6e
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/bdb.(*db).beginTx(0xc4204aa000, 0xc4204aa100, 0xc4204aa180, 0xc4200598f0, 0x43dacf)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/bdb/db.go:281 +0x38
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/bdb.(*db).BeginReadTx(0xc4204aa000, 0x7e8560, 0x3, 0xc4201dfc80, 0xc420069200)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/bdb/db.go:289 +0x30
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb.View(0xb7b9c0, 0xc4204aa000, 0xc420059968, 0x42b829, 0xc400000008)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/interface.go:192 +0x35
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/headerfs.(*headerIndex).chainTip(0xc420465120, 0x99a3a0, 0xc420068cc0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/headerfs/index.go:222 +0x7b
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/headerfs.(*BlockHeaderStore).ChainTip(0xc42020c068, 0x0, 0xb5fa650e00000000, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/headerfs/store.go:451 +0xef
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*blockManager).current(0xc4202087e0, 0xc42048a9c4)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blockmanager.go:693 +0x36
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*blockManager).handleHeadersMsg(0xc4202087e0, 0xc4202824d0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blockmanager.go:1183 +0x16c0
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*blockManager).blockHandler(0xc4202087e0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blockmanager.go:347 +0x45c
created by github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*blockManager).Start
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/blockmanager.go:200 +0xe5
goroutine 165 [select]:
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer.(*Peer).stallHandler(0xc4204a8000)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer/peer.go:1324 +0x2b3
created by github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer/peer.go:2062 +0x2cb
goroutine 166 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7f20cf8abbc0, 0x72, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420212098, 0x72, 0xffffffffffffff00, 0xb77f80, 0xb74548)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420212098, 0xc420192000, 0x18, 0x18)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420212080, 0xc420192060, 0x18, 0x18, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc420212080, 0xc420192060, 0x18, 0x18, 0x8, 0xc4202439e8, 0x57df7e)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc42000e008, 0xc420192060, 0x18, 0x18, 0x0, 0x0, 0x0)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/net/net.go:176 +0x6d
io.ReadAtLeast(0x7f20cf90dee0, 0xc42000e008, 0xc420192060, 0x18, 0x18, 0x18, 0x7bec00, 0xc420236001, 0xc420192060)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/io/io.go:309 +0x86
io.ReadFull(0x7f20cf90dee0, 0xc42000e008, 0xc420192060, 0x18, 0x18, 0x7f20cf908d18, 0xc420243b10, 0x590513)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/io/io.go:327 +0x58
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/wire.readMessageHeader(0x7f20cf90dee0, 0xc42000e008, 0xc4202f81c0, 0x20001117d, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/wire/message.go:196 +0x7e
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/wire.ReadMessageWithEncodingN(0x7f20cf90dee0, 0xc42000e008, 0x12141c160001117d, 0x7f2000000002, 0xc42000e008, 0x59e94616, 0x3, 0x0, 0x1, 0xc420243de0, ...)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/wire/message.go:332 +0x50
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer.(*Peer).readMessage(0xc4204a8000, 0x4500000002, 0xbb5b00, 0xc420236050, 0xc420236058, 0x8, 0x8, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer/peer.go:1139 +0xb3
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer.(*Peer).inHandler(0xc4204a8000)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer/peer.go:1461 +0xd9
created by github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer/peer.go:2063 +0x2f0
goroutine 142 [select, 9 minutes]:
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/addrmgr.(*AddrManager).addressHandler(0xc420436a00)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/addrmgr/addrmanager.go:342 +0x143
created by github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/addrmgr.(*AddrManager).Start
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/addrmgr/addrmanager.go:551 +0xfb
goroutine 141 [select, 9 minutes]:
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).peerHandler(0xc4203fc000)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/neutrino.go:911 +0x3cd
created by github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).Start
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/neutrino.go:1250 +0x78
goroutine 163 [chan receive, 9 minutes]:
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer.(*Peer).WaitForDisconnect(...)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer/peer.go:2078
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).peerDoneHandler(0xc4203fc000, 0xc420082580)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/neutrino.go:1213 +0x43
created by github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).outboundPeerConnected
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/neutrino.go:1206 +0x396
goroutine 167 [select]:
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer.(*Peer).queueHandler(0xc4204a8000)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer/peer.go:1704 +0x313
created by github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer/peer.go:2064 +0x315
goroutine 168 [select, 3 minutes]:
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer.(*Peer).outHandler(0xc4204a8000)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer/peer.go:1838 +0x14f
created by github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer/peer.go:2065 +0x33a
goroutine 170 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc4204aa190)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/sema.go:56 +0x39
sync.(*RWMutex).Lock(0xc4204aa188)
	/home/travis/.gimme/versions/go1.9.linux.amd64/src/sync/rwmutex.go:98 +0x6e
github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*DB).mmap(0xc4204aa000, 0x11000, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/db.go:246 +0x5b
github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*DB).allocate(0xc4204aa000, 0x1, 0x0, 0x0, 0xc4202b4f50)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/db.go:849 +0x12f
github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*Tx).allocate(0xc4203f48c0, 0x1, 0x7f20cf893000, 0xbd6540, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/tx.go:456 +0x3d
github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*node).spill(0xc4202b4f50, 0xc42022b710, 0xc420248d58)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/node.go:368 +0x274
github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc420069080, 0xc42022b600, 0xc420248fd0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/bucket.go:570 +0x4d3
github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc420069040, 0xc42022b600, 0xc420249248)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/bucket.go:537 +0x417
github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc4203f48d8, 0x1514e4ce2, 0xbb5be0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/bucket.go:537 +0x417
github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*Tx).Commit(0xc4203f48c0, 0xb7db80, 0xc420069080)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/tx.go:163 +0x129
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/bdb.(*transaction).Commit(0xc42020c458, 0xc42020c458, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/bdb/db.go:91 +0x32
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb.Update(0xb7b9c0, 0xc4204aa000, 0xc4202494d0, 0x729d38, 0x1)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/interface.go:225 +0xce
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/filterdb.(*FilterStore).PutFilter(0xc42044c160, 0xc42001cde0, 0xc42022b5f0, 0xc420249601, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/filterdb/db.go:150 +0x6a
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).GetCFilter(0xc4203fc000, 0x873d71d7281caed7, 0xe5a20b156a4d4241, 0x7cb0bd6ef27b3bdf, 0x27322f7498fc18f0, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/query.go:449 +0x50f
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).blockFilterMatches(0xc4203fc000, 0xc420198000, 0xc4201d6244, 0x873d71d7281caed7, 0xe5a20b156a4d4241, 0x7cb0bd6ef27b3bdf)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:551 +0x1ff
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).notifyBlock(0xc4203fc000, 0xc420198000, 0xc4202f9260, 0xc4201d6240, 0x1, 0x2, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:457 +0x10d
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).Rescan(0xc4203fc000, 0xc420069280, 0x5, 0x8, 0x0, 0x0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:348 +0x1a9e
github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*Rescan).Start.func1(0xc42009c180, 0xc42020a8a0)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:747 +0x9e
created by github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*Rescan).Start
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:745 +0x8d
goroutine 169 [select, 3 minutes]:
github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer.(*Peer).pingHandler(0xc4204a8000)
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer/peer.go:1912 +0x12e
created by github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer.(*Peer).start
	/home/travis/gopath/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcd/peer/peer.go:2066 +0x35f
rax    0xfffffffffffffffc
rbx    0x3b8e273f
rcx    0x45ec33
rdx    0x0
rdi    0xbb5a40
rsi    0x0
rbp    0xc420035ee8
rsp    0xc420035ea0
r8     0x0
r9     0x0
r10    0xc420035ed8
r11    0x206
r12    0x1
r13    0xc420024780
r14    0x1
r15    0x0
rip    0x45ec33
rflags 0x206
cs     0x33
fs     0x0
gs     0x0

Improve tests

Currently, tests are monolithic and include what should actually be a benchmark. They should be broken up into bite-sized pieces and we should add real benchmarking. We should also add integration tests for stopping and restarting the ChainService at various points during the synchronization and make sure it restarts correctly.

go get doesn't work

 go get -v github.com/lightninglabs/neutrino
 github.com/lightninglabs/neutrino (download)
 github.com/lightninglabs/neutrino
 ../go/src/github.com/lightninglabs/neutrino/blockmanager.go:651:43: undefined: wire.FilterType
 ../go/src/github.com/lightninglabs/neutrino/blockmanager.go:772:43: undefined: wire.FilterType
 ../go/src/github.com/lightninglabs/neutrino/blockmanager.go:1020:47: undefined: wire.MsgCFHeaders
 ../go/src/github.com/lightninglabs/neutrino/blockmanager.go:1139:13: undefined: wire.MsgCFHeaders
 ../go/src/github.com/lightninglabs/neutrino/blockmanager.go:1160:43: undefined: wire.FilterType
 ../go/src/github.com/lightninglabs/neutrino/blockmanager.go:1301:51: undefined: wire.MsgCFHeaders
 ../go/src/github.com/lightninglabs/neutrino/blockmanager.go:1329:58: undefined: wire.FilterType
 ../go/src/github.com/lightninglabs/neutrino/blockmanager.go:1424:8: undefined: wire.FilterType
 ../go/src/github.com/lightninglabs/neutrino/blockmanager.go:1424:37: undefined: wire.MsgCFHeaders
 ../go/src/github.com/lightninglabs/neutrino/blockmanager.go:1474:13: undefined: wire.FilterType
 ../go/src/github.com/lightninglabs/neutrino/blockmanager.go:1474:13: too many errors

consistently crash on simnet when restarting after some blocks were mined

This issue is about a crash we having and we managed create a reproducer on simnet.
I have a setup of:

• 2 Lnd nodes updated to master
• 1 Btcd node configured to work with compact filters updated to master.

Both of them configured to work on simnet.

I am then executing the following steps:

1. run all nodes
2. mine 400 blocks
3. open channel from one lnd node to another
4. generate 6 blocks for channel to open
5. stop lnd node
6. mine one block
7. restart lnd

Result:
lnd crashes consistently on every restart. Only after I mine 1000 blocks is sometimes recover and runs.

This is the log for the first restart crash
first_restart.txt

And this one is for the second restart (it is different a bit)
second_restart.txt

Sync interruption due to header difficulty

Multiple users are reporting persistent errors similar to the following:

lnd: 2017-10-20 03:29:18.639 [INF] BTCN: Syncing to block height 1210641 from peer 165.227.7.29:18333 2017-10-20 03:29:18.639 [INF] BTCN: Downloading headers for blocks 930001 to 1000007 from peer 165.227.7.29:18333
lnd: 2017-10-20 03:29:19.124 [WRN] BTCN: Header doesn't pass sanity check: block target difficulty of 00000000ffff0000000000000000000000000000000000000000000000000000 is higher than max of 0000000000000202f00000000000000000000000000000000000000000000000 -- disconnecting peer 2017-10-20 03:29:19.124 [INF] BTCN: Lost peer 165.227.7.29:18333 (outbound)
lnd: 2017-10-20 03:29:19.124 [WRN] BTCN: No sync peer candidates available
lnd: 2017-10-20 03:29:24.308 [INF] BTCN: New valid peer 165.227.7.29:18333 (outbound) (/btcwire:0.5.0/btcd:0.12.0/)
lnd: 2017-10-20 03:29:24.309 [INF] BTCN: Syncing to block height 1210641 from peer 165.227.7.29:18333 2017-10-20 03:29:24.309 [INF] BTCN: Downloading headers for blocks 930001 to 1000007 from peer 165.227.7.29:18333
lnd: 2017-10-20 03:29:24.813 [WRN] BTCN: Header doesn't pass sanity check: block target difficulty of 00000000ffff0000000000000000000000000000000000000000000000000000 is higher than max of 0000000000000202f00000000000000000000000000000000000000000000000 -- disconnecting peer 2017-10-20 03:29:24.813 [INF] BTCN: Lost peer 165.227.7.29:18333 (outbound)
lnd: 2017-10-20 03:29:24.813 [WRN] BTCN: No sync peer candidates available

Investigate and add support for Private Information Retrieval for private block and possibly cfilter fetching

An implementation of PIR lives at http://percy.sourceforge.net/ - we should investigate what it would take to add support for this mechanism to fetch blocks with better privacy. We should also investigate using this for cfilter retrieval as otherwise we can reveal to peers when our wallet may have been created.

rescan: update filter fetching interfaces to allow for batch filter fetching

The finalized version of BIP 157 allows nodes to query for a batch of filters in a single message. This is useful as if we're catching up from a few thousand blocks, we can request them all in a single message, and get the responses pipelined, rather than the back and forth of repeatedly fetching each filter individually.

Currently, this isn't exposed to the caller, nor utilized internally for rescan (or GetUtxo) purposes.

Steps To Completion

• Modify the GetCfilters method to expose both a start height and stop hash just as in the p2p message. To start with, one can convert all the existing calls to use the height of the filter to be requested, and use the stop hash of that very same header.

• Where relevant (in the rescan loop for example) start fetching filters in batch, rather than fetching them individually. One way to do this is at the top of the loop we can check if our starting height is very far behind the height of the main node. If so, we can likely fetch the filters using a single batch query.

• Update the GetUTXO method to also fetch the filters in batch. This is likely where we'll see an even bigger savings as outside of the regular rescan loop, we may be starting far-ish in the past.

The above changes will likely also affect btcsuite/btcwallet and a similar set of optimizations during it's initial chain sync, and also the HD seed import rescan can take advantage of this optimization.

go get doesn't work on Debian 9

On a fresh detain 9 box with go 1.11.1:

$ go get -v github.com/lightninglabs/neutrino
github.com/lightninglabs/neutrino (download)
created GOPATH=/home/varunram/go; see 'go help gopath'
github.com/btcsuite/btcd (download)
github.com/btcsuite/btclog (download)
github.com/btcsuite/btcutil (download)
Fetching https://golang.org/x/crypto/ripemd160?go-get=1
Parsing meta tags from https://golang.org/x/crypto/ripemd160?go-get=1 (status code 200)
get "golang.org/x/crypto/ripemd160": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https:
//golang.org/x/crypto/ripemd160?go-get=1
get "golang.org/x/crypto/ripemd160": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
golang.org/x/crypto (download)
github.com/btcsuite/go-socks (download)
github.com/davecgh/go-spew (download)
github.com/btcsuite/websocket (download)
github.com/aead/siphash (download)
github.com/kkdai/bstream (download)
github.com/btcsuite/btcwallet (download)
github.com/btcsuite/golangcrypto (download)
github.com/lightninglabs/neutrino
# github.com/lightninglabs/neutrino
go/src/github.com/lightninglabs/neutrino/neutrino.go:1205:4: cannot use sp.OnVersion (type func(*peer.Peer, *wire.MsgVersion)) as type func(*peer.Peer, *wire.MsgVer
sion) *wire.MsgReject in field value

Boltdb panic during initial chain sync

Expected behavior:

Start a fresh lnd node, point at suitable neutrino forked btcd node and sync chain to tip.

Actual behavior:

Lnd crashed after roughly 500k blocks due to panic from boltdb. From issues linked below this seems to be a race condition with respect to boltdb access.

lnd_1       | 2018-02-21 02:43:35.649 [DBG] BTCN: Sending getcfilter to 192.168.0.104:8333 (outbound)
lnd_1       | 2018-02-21 02:43:35.654 [INF] LTND: LightningWallet opened
lnd_1       | 2018-02-21 02:43:35.666 [INF] HSWC: Starting HTLC Switch
lnd_1       | 2018-02-21 02:43:35.671 [INF] RPCS: RPC server listening on 127.0.0.1:10009
lnd_1       | panic: page 25606 already freed
lnd_1       |
lnd_1       | goroutine 57 [running]:
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*freelist).free(0xc4204e8540, 0x1fa4a, 0x7fbd46406000)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/freelist.go:121 +0x29a
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*node).spill(0xc4204dbdc0, 0xc42014b300, 0x14ebcc0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/node.go:363 +0x1e0
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*node).spill(0xc4204dbd50, 0xc42014b2e0, 0x14ebcc0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/node.go:350 +0xb4
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*node).spill(0xc4204dbc70, 0xc4202710e0, 0xc4203f2d40)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/node.go:350 +0xb4
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc4204f2700, 0xc420271000, 0xc4203f2fb0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/bucket.go:570 +0x17b
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc4204f26c0, 0xc420270f00, 0xc4203f3220)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/bucket.go:537 +0x8f8
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*Bucket).spill(0xc4201b9518, 0x284007a3, 0x1538920)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/bucket.go:537 +0x8f8
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt.(*Tx).Commit(0xc4201b9500, 0xa, 0x14f3620)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/boltdb/bolt/tx.go:163 +0x125
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/bdb.(*transaction).Commit(0xc4204f0088, 0xc4204f0088, 0x0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/bdb/db.go:91 +0x32
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb.Update(0x14eddc0, 0xc4201421e0, 0xc4203f34a8, 0x87caee, 0x180001)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/roasbeef/btcwallet/walletdb/interface.go:225 +0xd1
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/filterdb.(*FilterStore).PutFilter(0xc420086000, 0xc42014ab80, 0xc420270ea0, 0xc4203f3600, 0x0, 0x0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/filterdb/db.go:170 +0x77
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).GetCFilter(0xc42042a780, 0x485af27c8018b071, 0x97193b0ae97a4b61, 0x57f8057bd2a24, 0x0, 0x57f8057bd2a00, 0x0, 0x0, 0x0, 0x0, ...)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/query.go:450 +0x527
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).blockFilterMatches(0xc42042a780, 0xc4204fe780, 0xc420270664, 0x0, 0x7cf25a4871b01880, 0xa3b1997614b7ae9)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:530 +0x138
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).notifyBlock(0xc42042a780, 0xc4204fe780, 0xc4204db730, 0xc420270660, 0x1, 0x2, 0x0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:452 +0x145
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*ChainService).Rescan(0xc42042a780, 0xc4204f50e0, 0x7, 0xc, 0x0, 0x0)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:432 +0xeff
lnd_1       | github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*Rescan).Start.func1(0xc4204f4fc0, 0xc4204f5080)
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:741 +0xb7
lnd_1       | created by github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino.(*Rescan).Start
lnd_1       | 	/go/src/github.com/lightningnetwork/lnd/vendor/github.com/lightninglabs/neutrino/rescan.go:751 +0x8d

Affects:
LND commit 2a61ccec
BTCD commit 0dcaa7f9
Running on Ubuntu 16.04 (in Docker)

More detailed log excepts can be found here

Related boltdb issues
boltdb/bolt#731
etcd-io/bbolt#72

Note: this occurs only on mainnet and not on testnet

[Question] How does `lnd` use Neutrino?

Is it a Go package/library or I can access a Neutrino process from a different language through some kind of inter process communication?

Add keepalives to prevent disconnection from peers while waiting for events

Currently there's a 45-second timeout somewhere and it should be fixed.

feature request: add litecoin support

Neutrino currently does not support litecoin network, in either mainnet or testnet. It is frustrating to build litecoin software without even testnet support.

Change most of blockmanager to use the `queryPeers` API

Currently, there's a lot of state to keep track of in the block manager. This would be much simpler and more maintainable if all the queries against peers were done using the query API. This also requires some extensions to the query API, specifically the ability to query all peers at once for cfheaders.

Optimize database implementation

The current database implementation is very naive: it uses Bolt for everything. Instead, all headers (block headers and cfheaders) should be stored in flat files and only metadata and cfilters themselves should be stored in Bolt.

Build errors from BuildBasicFilter

I just checked out out master, got dependencies and tried to build with go build. I received these errors:

> go build
# github.com/lightninglabs/neutrino/filterdb
../../lightninglabs/neutrino/filterdb/db.go:102:47: not enough arguments in call to builder.BuildBasicFilter
        have (*wire.MsgBlock)
        want (*wire.MsgBlock, [][]byte)
../../lightninglabs/neutrino/filterdb/db.go:119:21: undefined: builder.BuildExtFilter
../../lightninglabs/neutrino/filterdb/db.go:211:34: not enough arguments in call to gcs.FromNBytes
        have (number, []byte)
        want (uint8, uint64, []byte)
# github.com/lightninglabs/neutrino/headerfs
../../lightninglabs/neutrino/headerfs/store.go:529:48: not enough arguments in call to builder.BuildBasicFilter
        have (*wire.MsgBlock)
        want (*wire.MsgBlock, [][]byte)
../../lightninglabs/neutrino/headerfs/store.go:545:22: undefined: builder.BuildExtFilter

Did btcutil have an update?