Comments (22)
jolan
commented on June 16, 2024
kill -ABRT and paste the traceback please.
from dcrstakepool.
chappjc
commented on June 16, 2024
Link removed. According to @cjepson it was hung on:
github.com/decred/dcrrpcclient.FutureGetStakeInfoResult.Receive(0xc820400ae0, 0xc820400ae0, 0x0, 0x0)
from dcrstakepool.
chappjc
commented on June 16, 2024
One of the wallet processes was suspended, by tmux probably. The stake pool was hung on getstakeinfo. Somewhere ahead of this. (Be sure to exit scrollback mode before detaching or suffer the consequences.)
When the process came back to life and resynced, dcrstakepool finally printed messages and stopped:
[INF] DCRS: getStakeInfoFn failure on server 0: -4: the wallet is currently syncing to the best block, please try again later
[INF] DCRS: RPC GetStakeInfo failed: -4: the wallet is currently syncing to the best block, please try again later
Goji received signal, gracefully stopping
I think at least 3 things are needed here:
- A timeout so that it doesn't hang indefinitely if the wallet is stuck for any reason. (but getbalance and other commands did respond, although getstakeinfo had the error). I'd be happy to close the issue if just this.
- More info in /status, like output from different commands. In this case it just said "RPC Status: Running"
- EDIT: Verbose logging mode
On the upside, our testnet sandbox pool voted 3 times and all is well.
from dcrstakepool.
jyap808
commented on June 16, 2024
I have the same issue. Here is the trace.
2016/04/21 22:10:12.841261 Starting Goji on [::]:8000
22:10:12 2016-04-21 [INF] DCRS: Starting wallet RPC manager
SIGABRT: abort
PC=0x461e71 m=0
goroutine 11 [syscall, 16 minutes]:
runtime.notetsleepg(0x115ad00, 0xffffffffffffffff, 0x1)
/usr/local/go/src/runtime/lock_futex.go:205 +0x4e fp=0xc820022f40 sp=0xc820022f18
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x132 fp=0xc820022f78 sp=0xc820022f40
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x18 fp=0xc820022fc0 sp=0xc820022f78
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820022fc8 sp=0xc820022fc0
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x37
goroutine 1 [IO wait, 2 minutes]:
net.runtime_pollWait(0x7f8038f34f68, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820226450, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820226450, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8202263f0, 0x0, 0x7f8038f4fff8, 0xc82028e000)
/usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc82002a6f8, 0xc820018100, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
github.com/zenazn/goji/graceful.tcpKeepAliveListener.Accept(0xc82002a6f8, 0x0, 0x0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/zenazn/goji/graceful/server.go:21 +0x41
github.com/zenazn/goji/graceful.(*gracefulServer).Accept(0xc8201c85c0, 0x0, 0x0, 0x0, 0x0)
<autogenerated>:1 +0x6b
github.com/zenazn/goji/graceful/listener.(*T).Accept(0xc8203d5040, 0x0, 0x0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/zenazn/goji/graceful/listener/listener.go:75 +0x71
net/http.(*Server).Serve(0xc820019a80, 0x7f8038f4ffc0, 0xc8203d5040, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2117 +0x129
github.com/zenazn/goji/graceful.(*Server).Serve(0xc820103c10, 0x7f8038f4feb0, 0xc8201c85c0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/zenazn/goji/graceful/serve13.go:74 +0x1c6
github.com/zenazn/goji/graceful.Serve(0x7f8038f4fe78, 0xc82002a6f8, 0x7f8038f4fe50, 0xc8203b8f30, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/zenazn/goji/graceful/server.go:107 +0xfc
github.com/zenazn/goji.ServeListener(0x7f8038f4fda0, 0xc82002a6f8)
/REDACTED/projects/gopath/src/github.com/zenazn/goji/serve.go:57 +0x25c
github.com/zenazn/goji.Serve()
/REDACTED/projects/gopath/src/github.com/zenazn/goji/serve.go:31 +0x4f
main.main()
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/server.go:115 +0x196d
goroutine 17 [syscall, 16 minutes, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1
goroutine 5 [semacquire, 16 minutes]:
sync.runtime_Syncsemacquire(0xc82001d7d0)
/usr/local/go/src/runtime/sema.go:241 +0x201
sync.(*Cond).Wait(0xc82001d7c0)
/usr/local/go/src/sync/cond.go:63 +0x9b
github.com/btcsuite/seelog.(*asyncLoopLogger).processItem(0xc8200587e0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/seelog/behavior_asynclooplogger.go:50 +0xc7
github.com/btcsuite/seelog.(*asyncLoopLogger).processQueue(0xc8200587e0)
/REDACTED/projects/gopath/src/github.com/btcsuite/seelog/behavior_asynclooplogger.go:63 +0x2a
created by github.com/btcsuite/seelog.newAsyncLoopLogger
/REDACTED/projects/gopath/src/github.com/btcsuite/seelog/behavior_asynclooplogger.go:40 +0x91
goroutine 6 [semacquire, 16 minutes]:
sync.runtime_Syncsemacquire(0xc82001d950)
/usr/local/go/src/runtime/sema.go:241 +0x201
sync.(*Cond).Wait(0xc82001d940)
/usr/local/go/src/sync/cond.go:63 +0x9b
github.com/btcsuite/seelog.(*asyncLoopLogger).processItem(0xc820058900, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/seelog/behavior_asynclooplogger.go:50 +0xc7
github.com/btcsuite/seelog.(*asyncLoopLogger).processQueue(0xc820058900)
/REDACTED/projects/gopath/src/github.com/btcsuite/seelog/behavior_asynclooplogger.go:63 +0x2a
created by github.com/btcsuite/seelog.newAsyncLoopLogger
/REDACTED/projects/gopath/src/github.com/btcsuite/seelog/behavior_asynclooplogger.go:40 +0x91
goroutine 7 [chan receive, 2 minutes]:
github.com/golang/glog.(*loggingT).flushDaemon(0x113d120)
/REDACTED/projects/gopath/src/github.com/golang/glog/glog.go:882 +0x67
created by github.com/golang/glog.init.1
/REDACTED/projects/gopath/src/github.com/golang/glog/glog.go:410 +0x297
goroutine 10 [semacquire, 14 minutes]:
sync.runtime_Syncsemacquire(0xc8203d54d0)
/usr/local/go/src/runtime/sema.go:241 +0x201
sync.(*Cond).Wait(0xc8203d54c0)
/usr/local/go/src/sync/cond.go:63 +0x9b
github.com/btcsuite/seelog.(*asyncAdaptiveLogger).processItem(0xc820018300, 0xc820412c00, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/seelog/behavior_adaptivelogger.go:94 +0xdd
github.com/btcsuite/seelog.(*asyncAdaptiveLogger).processQueue(0xc820018300)
/REDACTED/projects/gopath/src/github.com/btcsuite/seelog/behavior_adaptivelogger.go:119 +0x2a
created by github.com/btcsuite/seelog.newAsyncAdaptiveLogger
/REDACTED/projects/gopath/src/github.com/btcsuite/seelog/behavior_adaptivelogger.go:84 +0x51e
goroutine 12 [chan receive, 16 minutes]:
github.com/zenazn/goji/graceful.sigLoop()
/REDACTED/projects/gopath/src/github.com/zenazn/goji/graceful/signal.go:126 +0x4c
created by github.com/zenazn/goji/graceful.init.2
/REDACTED/projects/gopath/src/github.com/zenazn/goji/graceful/signal.go:121 +0x2b
goroutine 14 [chan receive, 16 minutes]:
database/sql.(*DB).connectionOpener(0xc82042c000)
/usr/local/go/src/database/sql/sql.go:727 +0x45
created by database/sql.Open
/usr/local/go/src/database/sql/sql.go:493 +0x33f
goroutine 31 [select, 16 minutes]:
github.com/decred/dcrrpcclient.(*Client).wsReconnectHandler(0xc82007c1e0)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:633 +0x871
created by github.com/decred/dcrrpcclient.New
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:1283 +0x652
goroutine 30 [select, 2 minutes]:
github.com/decred/dcrrpcclient.(*Client).wsOutHandler(0xc82007c1e0)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:429 +0x31d
created by github.com/decred/dcrrpcclient.(*Client).start
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:1052 +0x1ec
goroutine 29 [IO wait, 2 minutes]:
net.runtime_pollWait(0x7f8038f35268, 0x72, 0xc8201ca000)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820416610, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820416610, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8204165b0, 0xc8201ca000, 0x2000, 0x2000, 0x0, 0x7f8038f26050, 0xc82000a048)
/usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82002a4b0, 0xc8201ca000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc82049ea20, 0x7f8038f39c40, 0xc82002a4b0, 0x5, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc820096600, 0xd89b17, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc820096600, 0xc8203c4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:939 +0x167
bufio.(*Reader).fill(0xc8200e8660)
/usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc8200e8660, 0xc8201be868, 0x2, 0x8, 0xc820414c00, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:207 +0x260
github.com/btcsuite/websocket.(*Conn).readFull(0xc82007e2d0, 0xc8201be868, 0x2, 0x8, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:542 +0xa9
github.com/btcsuite/websocket.(*Conn).advanceFrame(0xc82007e2d0, 0x409245, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:566 +0x159
github.com/btcsuite/websocket.(*Conn).NextReader(0xc82007e2d0, 0xe0, 0x0, 0x0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:706 +0x76
github.com/btcsuite/websocket.(*Conn).ReadMessage(0xc82007e2d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:769 +0x55
github.com/decred/dcrrpcclient.(*Client).wsInHandler(0xc82007c1e0)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:393 +0x1ce
created by github.com/decred/dcrrpcclient.(*Client).start
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:1051 +0x1ca
goroutine 44 [chan receive, 2 minutes]:
github.com/decred/dcrrpcclient.FutureGetStakeInfoResult.Receive(0xc8201bc4e0, 0xc8201bc4e0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/wallet.go:2871 +0x87
github.com/decred/dcrrpcclient.(*Client).GetStakeInfo(0xc82007c3c0, 0xc820228230, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/wallet.go:2899 +0x3b
github.com/decred/dcrstakepool/controllers.(*walletSvrManager).executeInSequence(0xc82005e1b0, 0x9, 0xafee40, 0xc8201bc420, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/controllers/dcrclient.go:539 +0x50a7
github.com/decred/dcrstakepool/controllers.(*walletSvrManager).walletRPCHandler(0xc82005e1b0)
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/controllers/dcrclient.go:241 +0xf61
created by github.com/decred/dcrstakepool/controllers.(*walletSvrManager).Start
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/controllers/dcrclient.go:906 +0x161
goroutine 33 [IO wait, 12 minutes]:
net.runtime_pollWait(0x7f8038f351a8, 0x72, 0xc8201c5800)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8204167d0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8204167d0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820416770, 0xc8201c5800, 0x800, 0x800, 0x0, 0x7f8038f26050, 0xc82000a048)
/usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82002a040, 0xc8201c5800, 0x800, 0x800, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc82038c930, 0x7f8038f39c40, 0xc82002a040, 0x5, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc820096c00, 0xd89b17, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc820096c00, 0xc820098000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:939 +0x167
bufio.(*Reader).fill(0xc820058180)
/usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc820058180, 0xc8201bfdf0, 0x2, 0x8, 0xc82028a000, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:207 +0x260
github.com/btcsuite/websocket.(*Conn).readFull(0xc82041e000, 0xc8201bfdf0, 0x2, 0x8, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:542 +0xa9
github.com/btcsuite/websocket.(*Conn).advanceFrame(0xc82041e000, 0x409245, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:566 +0x159
github.com/btcsuite/websocket.(*Conn).NextReader(0xc82041e000, 0x800, 0x0, 0x0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:706 +0x76
github.com/btcsuite/websocket.(*Conn).ReadMessage(0xc82041e000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:769 +0x55
github.com/decred/dcrrpcclient.(*Client).wsInHandler(0xc82007c3c0)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:393 +0x1ce
created by github.com/decred/dcrrpcclient.(*Client).start
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:1051 +0x1ca
goroutine 34 [select, 2 minutes]:
github.com/decred/dcrrpcclient.(*Client).wsOutHandler(0xc82007c3c0)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:429 +0x31d
created by github.com/decred/dcrrpcclient.(*Client).start
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:1052 +0x1ec
goroutine 35 [select, 15 minutes]:
github.com/decred/dcrrpcclient.(*Client).wsReconnectHandler(0xc82007c3c0)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:633 +0x871
created by github.com/decred/dcrrpcclient.New
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:1283 +0x652
goroutine 37 [IO wait, 12 minutes]:
net.runtime_pollWait(0x7f8038f350e8, 0x72, 0xc820482800)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820422370, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820422370, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820422310, 0xc820482800, 0x800, 0x800, 0x0, 0x7f8038f26050, 0xc82000a048)
/usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82002a068, 0xc820482800, 0x800, 0x800, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc820012750, 0x7f8038f39c40, 0xc82002a068, 0x5, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc820097200, 0xd89b17, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc820097200, 0xc820064000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:939 +0x167
bufio.(*Reader).fill(0xc820059380)
/usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc820059380, 0xc8201bff88, 0x2, 0x8, 0xc820208000, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:207 +0x260
github.com/btcsuite/websocket.(*Conn).readFull(0xc82041e0f0, 0xc8201bff88, 0x2, 0x8, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:542 +0xa9
github.com/btcsuite/websocket.(*Conn).advanceFrame(0xc82041e0f0, 0x409245, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:566 +0x159
github.com/btcsuite/websocket.(*Conn).NextReader(0xc82041e0f0, 0x800, 0x0, 0x0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:706 +0x76
github.com/btcsuite/websocket.(*Conn).ReadMessage(0xc82041e0f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:769 +0x55
github.com/decred/dcrrpcclient.(*Client).wsInHandler(0xc82007d040)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:393 +0x1ce
created by github.com/decred/dcrrpcclient.(*Client).start
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:1051 +0x1ca
goroutine 38 [select, 12 minutes]:
github.com/decred/dcrrpcclient.(*Client).wsOutHandler(0xc82007d040)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:429 +0x31d
created by github.com/decred/dcrrpcclient.(*Client).start
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:1052 +0x1ec
goroutine 39 [select, 15 minutes]:
github.com/decred/dcrrpcclient.(*Client).wsReconnectHandler(0xc82007d040)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:633 +0x871
created by github.com/decred/dcrrpcclient.New
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:1283 +0x652
goroutine 41 [IO wait, 12 minutes]:
net.runtime_pollWait(0x7f8038f35028, 0x72, 0xc820483000)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820422530, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820422530, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8204224d0, 0xc820483000, 0x800, 0x800, 0x0, 0x7f8038f26050, 0xc82000a048)
/usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82002a130, 0xc820483000, 0x800, 0x800, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc820012ff0, 0x7f8038f39c40, 0xc82002a130, 0x5, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc820097800, 0xd89b17, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc820097800, 0xc8204cc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:939 +0x167
bufio.(*Reader).fill(0xc8201bd0e0)
/usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc8201bd0e0, 0xc82047e1c0, 0x2, 0x8, 0xc8201b6000, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:207 +0x260
github.com/btcsuite/websocket.(*Conn).readFull(0xc82046c000, 0xc82047e1c0, 0x2, 0x8, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:542 +0xa9
github.com/btcsuite/websocket.(*Conn).advanceFrame(0xc82046c000, 0x409245, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:566 +0x159
github.com/btcsuite/websocket.(*Conn).NextReader(0xc82046c000, 0x800, 0x0, 0x0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:706 +0x76
github.com/btcsuite/websocket.(*Conn).ReadMessage(0xc82046c000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/btcsuite/websocket/conn.go:769 +0x55
github.com/decred/dcrrpcclient.(*Client).wsInHandler(0xc82007ce60)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:393 +0x1ce
created by github.com/decred/dcrrpcclient.(*Client).start
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:1051 +0x1ca
goroutine 42 [select, 12 minutes]:
github.com/decred/dcrrpcclient.(*Client).wsOutHandler(0xc82007ce60)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:429 +0x31d
created by github.com/decred/dcrrpcclient.(*Client).start
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:1052 +0x1ec
goroutine 43 [select, 14 minutes]:
github.com/decred/dcrrpcclient.(*Client).wsReconnectHandler(0xc82007ce60)
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:633 +0x871
created by github.com/decred/dcrrpcclient.New
/REDACTED/projects/gopath/src/github.com/decred/dcrrpcclient/infrastructure.go:1283 +0x652
goroutine 45 [select, 14 minutes, locked to thread]:
runtime.gopark(0xd89b48, 0xc8200f8f28, 0xbf6150, 0x6, 0x18, 0x2)
/usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc8200f8f28, 0x0, 0x18)
/usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc8200f8f28)
/usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1
goroutine 58 [chan receive, 2 minutes]:
github.com/decred/dcrstakepool/controllers.(*walletSvrManager).getStakeInfo(0xc82005e1b0, 0xd89bf8, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/controllers/dcrclient.go:828 +0x17a
github.com/decred/dcrstakepool/controllers.(*walletSvrManager).GetStakeInfo(0xc82005e1b0, 0x0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/controllers/dcrclient.go:848 +0x8d
github.com/decred/dcrstakepool/controllers.(*MainController).Stats(0xc82047fa10, 0x0, 0xc820158270, 0xc8200957a0, 0x0, 0x0, 0x0)
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/controllers/main.go:368 +0x51b
reflect.callMethod(0xc8201589c0, 0xc820296d38)
/usr/local/go/src/reflect/value.go:631 +0x1fc
reflect.methodValueCall(0x0, 0xc820158270, 0xc8200957a0, 0xabf400, 0xc8201589c0, 0xbc7d60, 0xc82047fa10, 0x4613, 0xf, 0xc8203b5180, ...)
/usr/local/go/src/reflect/asm_amd64.s:29 +0x36
github.com/decred/dcrstakepool/system.(*Application).Route.func1(0x0, 0xc820158270, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/system/core.go:102 +0x250
github.com/zenazn/goji/web.handlerFuncWrap.ServeHTTPC(0xc82049f140, 0x0, 0xc820158270, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/zenazn/goji/web/handler.go:25 +0x4e
github.com/zenazn/goji/web.(*router).route(0xc82045b458, 0xc820158150, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/zenazn/goji/web/router.go:119 +0x1c0
github.com/zenazn/goji/web.(*mStack).newStack.func1(0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/zenazn/goji/web/middleware.go:88 +0x59
net/http.HandlerFunc.ServeHTTP(0xc82028e080, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/gorilla/context.ClearHandler.func1(0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/gorilla/context/context.go:141 +0x85
net/http.HandlerFunc.ServeHTTP(0xc82028e0a0, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/decred/dcrstakepool/system.(*Application).ApplyCsrfProtection.func1(0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/system/middleware.go:132 +0xce3
net/http.HandlerFunc.ServeHTTP(0xc820158180, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/decred/dcrstakepool/system.(*Application).ApplyIsXhr.func1(0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/system/middleware.go:72 +0x1ae
net/http.HandlerFunc.ServeHTTP(0xc82028e0c0, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/decred/dcrstakepool/system.(*Application).ApplyAuth.func1(0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/system/middleware.go:60 +0x59a
net/http.HandlerFunc.ServeHTTP(0xc82028e0e0, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/decred/dcrstakepool/system.(*Application).ApplyDbMap.func1(0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/system/middleware.go:41 +0x111
net/http.HandlerFunc.ServeHTTP(0xc8201581b0, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/decred/dcrstakepool/system.(*Application).ApplySessions.func1(0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/system/middleware.go:33 +0x14c
net/http.HandlerFunc.ServeHTTP(0xc820158210, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/decred/dcrstakepool/system.(*Application).ApplyTemplates.func1(0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/decred/dcrstakepool/system/middleware.go:23 +0x111
net/http.HandlerFunc.ServeHTTP(0xc820158240, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/zenazn/goji/web/middleware.AutomaticOptions.func1(0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/zenazn/goji/web/middleware/options.go:70 +0x143
net/http.HandlerFunc.ServeHTTP(0xc82028e100, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/zenazn/goji/web/middleware.Recoverer.func1(0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/zenazn/goji/web/middleware/recoverer.go:29 +0x113
net/http.HandlerFunc.ServeHTTP(0xc82028e120, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/zenazn/goji/web/middleware.RequestID.func1(0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/zenazn/goji/web/middleware/request_id.go:68 +0x324
net/http.HandlerFunc.ServeHTTP(0xc82028e140, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/zenazn/goji/web.(*cStack).ServeHTTP(0xc820158150, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/zenazn/goji/web/middleware.go:46 +0x5b
github.com/zenazn/goji/web.(*Mux).ServeHTTP(0xc82045b420, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/REDACTED/projects/gopath/src/github.com/zenazn/goji/web/mux.go:45 +0x5d
net/http.(*ServeMux).ServeHTTP(0xc8203b8f30, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/usr/local/go/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc820019a80, 0x7f8038f50120, 0xc8203d60d0, 0xc8200957a0)
/usr/local/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc820018100)
/usr/local/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2137 +0x44e
rax 0xca
rbx 0x0
rcx 0xffffffffffffffff
rdx 0x0
rdi 0x115ad00
rsi 0x0
rbp 0x1
rsp 0xc820022e98
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x8
r13 0xd85c30
r14 0x9
r15 0x8
rip 0x461e71
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
from dcrstakepool.
jolan
commented on June 16, 2024
This looks to be the same as @chappjc where getstakeinfo isn't returning.
If it's still in this state can you run dcrctl --wallet walletinfo and dcrctl --wallet getstakeinfo against the problematic server and paste the result?
from dcrstakepool.
chappjc
commented on June 16, 2024
My "workaround" that I posted in IRC:
- Edit:
cacheTimerStakeInfo = 1 * time.Minute - In a
tmux:watch -n 40 curl -s https://dcr.stakepool.net/stats > /dev/null 2>&1
In my deploy script I actually have this as the last command:
tmux new -d -P -s watcher "watch -n 40 'curl -s https://`hostname`/stats > /dev/null 2>&1 '"
But this doesn't seem to be necessary for me anymore. Dunno why. Perhaps it's because I also have this:
sudo /sbin/sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_intvl=60 net.ipv4.tcp_keepalive_probes=5
from dcrstakepool.
jyap808
commented on June 16, 2024
Yes, what this looks like is it's closing the RPC connection.
I'll implement some workarounds but maybe some sort of RPC ping can be implemented to keep the connection alive.
from dcrstakepool.
jyap808
commented on June 16, 2024
Modifying the tcp_keepalive settings didn't work for me.
The "cacheTimerStakeInfo and curl stats" works great for me. Nice and stable now.
Thanks @chappjc
from dcrstakepool.
chappjc
commented on June 16, 2024
Glad that helped. The keepalive settings didn't seem to help me either, but my new web server is somehow different from the sandbox machine I was using at the start.
from dcrstakepool.
jyap808
commented on June 16, 2024
My web server runs on Azure which has a public and private IP. That is possibly the issue. Either way, the RPC connections to the remote wallets stay up.
from dcrstakepool.
chappjc
commented on June 16, 2024
My servers all have the same 1-to-1 NAT between a public and private IP. The wallets have this too, which requires custom certs.
from dcrstakepool.
jolan
commented on June 16, 2024
Here's a diff which turns on trace level debugging for dcrrpcclient. It'd be very useful if you could run it with the stock code and see what dcrrpcclient is doing when you experience hangs. (There will be tons of output FYI)
diff --git a/config.go b/config.go
index 3d82b84..08346b8 100644
--- a/config.go
+++ b/config.go
@@ -405,6 +405,8 @@ func loadConfig() (*config, []string, error) {
initSeelogLogger(filepath.Join(cfg.LogDir, defaultLogFilename))
setLogLevels(defaultLogLevel)
+ cfg.DebugLevel = "trace"
+
// Parse, validate, and set debug log level(s).
if err := parseAndSetDebugLevels(cfg.DebugLevel); err != nil {
err := fmt.Errorf("%s: %v", funcName, err.Error())
diff --git a/server.go b/server.go
index 91cf8ab..e27e40e 100644
--- a/server.go
+++ b/server.go
@@ -11,6 +11,7 @@ import (
"github.com/golang/glog"
"github.com/gorilla/context"
+ "github.com/decred/dcrrpcclient"
"github.com/decred/dcrstakepool/controllers"
"github.com/decred/dcrstakepool/system"
@@ -64,6 +65,8 @@ func main() {
}
}()
+ dcrrpcclient.UseLogger(dcrstakepoolLog)
+
// Setup static files
static := web.New()
publicPath := application.Config.Get("general.public_path").(string)
from dcrstakepool.
jolan
commented on June 16, 2024
@houstonbofh @dmayne @Entity325 may be hitting this as well.
Trying to reproduce on EC2 but not able to thus far.
from dcrstakepool.
houstonbofh
commented on June 16, 2024
For the record, our web server is 1 to 1 NAT to the world, but all wallets are on an internal VPN based network to each other and are "local." (No NAT)
from dcrstakepool.
chappjc
commented on June 16, 2024
I'm not really sure the NAT part is the problem, so much as a stateful firewall somewhere losing state.
Regardless, I'm not having this problem anymore. I keep a watch on /stats anyway.
from dcrstakepool.
houstonbofh
commented on June 16, 2024
Good thing you have a developer for a firewall here.... The state table on my firewall would idle out in 30 minutes. I just raised that to 2.5 hours. I will let you know if behavior chaznges... But... It is not web communication causing this. It seems to be a break between the dcrstakepool software and the mysql database, and those are on the same box!
from dcrstakepool.
chappjc
commented on June 16, 2024
That's a different issue then. The traces pointed to a hang with RPC calls.
from dcrstakepool.
houstonbofh
commented on June 16, 2024
I can not say for sure, since I have yet to actually get trace logs. :) If this changes behavior, we will know. At least you have a good firewall guy with a very configurable firewall to test with. :)
And it does seem to have extended the time to error on our end. Based on that, I am thinking you are right and I was wrong about the mysql issues.
from dcrstakepool.
houstonbofh
commented on June 16, 2024
Extending the RPC timeout does help if it is set for 30 minutes. But, only too a point. My RPC timout on the firewall is 2.5 hours, and after 2 hours it was hung again. Do you want me to test with a 10 minute timeout to see if it actually triggers this?
from dcrstakepool.
jolan
commented on June 16, 2024
Yes if you can try to reproduce and get a trace log that would be extremely useful.
If you're running the latest code, just turn on RPC tracing:
https://github.com/decred/dcrstakepool-private/blob/master/sample-dcrstakepool.conf#L47
Otherwise you will need to add this line and recompile:
https://github.com/decred/dcrstakepool-private/blob/master/server.go#L68
I suspected that there was a reconnect bug where it tried to reconnect to the wrong IP and just kept timing out and re-trying but the reconnect code looks sane to me:
from dcrstakepool.
houstonbofh
commented on June 16, 2024
Once I get a chance to update all the software to current, I will do some testing on short timeouts!
from dcrstakepool.
jolan
commented on June 16, 2024
Closing since this is likely fixed by #55. What ended up happening is the RPC handler would do automatic reconnects and fail over and over which blocked the web page rendering which blocked the web server.
Now downed wallets are handled gracefully and wallets are manually reconnected.
from dcrstakepool.
Related Issues (20)
- Improve golint score HOT 3
- Transition from account-based to ticket-based system HOT 6
- Display pool fee address and percentage more prominently
- Lost redeem script HOT 6
- Design and API proposal for accountless operation HOT 5
- nginx limit_req delay HOT 1
- Show vote choices without the need to click HOT 1
- Home page: focus on rewards, not profits HOT 1
- Configurable certificate directory HOT 3
- enable simnet HOT 3
- I have several live tickets, they are not showing on the Tickets page HOT 11
- Open tickets block explorer links in new tab
- stakepoold: Make startup redeem script checking more efficient HOT 2
- rpc error: code = Unknown desc = -4: Method not found HOT 2
- VSP "working" flag in API HOT 1
- Failed to revoke tickets: -4:
- AddMissingTicket RPC failed
- README: Add notice about vspd
- Add deprecation notice and vspd link to UI home page
- Create tag and release for latest dcrstakepool
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dcrstakepool.