Promoted a configuration to a host that included memcache servers I had not built yet. Mcrouter started cleanly, however commands sent to mcrouter pool with the invalid hosts causes mcrouter to crash.
I would expect mcrouter to perhaps TKO the host immediately and keep retrying it to see if it comes back at some point; or just permanently mark it dead, requiring a mcrouter restart to bring it back online. Perhaps logging at start so users can see they have a node not resolving immediately?
This is using the latest version with the latest changes from April 29th. I have not tried this scenario before today.
Let me know if I can provide any more information. Here's logs, which I've thinned down a bit to make it easier to read.
I0506 15:32:26.450889 15456 McrouterInstance.cpp:581] reconfigured 1 proxies with 16 clients (111c75b2a583c734860392c8262cdb8d)
I0506 15:32:26.451611 15456 server.cpp:150] Spawning AsyncMcServer
I0506 15:33:12.448019 15463 EventBase.cpp:292] EventBase(): Starting loop.
I0506 15:33:12.448074 15463 AsyncSocket.cpp:211] new AsyncSocket(0x7f3fbc001740, evb=0x273cc58, fd=24)
I0506 15:33:12.448118 15463 AsyncSocket.cpp:513] AsyncSocket::setReadCallback() this=0x7f3fbc001740, fd=24, callback=0x7f3fbc001940, state=2
I0506 15:33:12.448132 15463 AsyncSocket.cpp:1694] AsyncSocket::updateEventRegistration(this=0x7f3fbc001740, fd=24, evb=0x273cc58, state=2, events=2
I0506 15:33:12.448151 15463 EventBase.cpp:421] EventBase(): Done with loop.
I0506 15:33:12.448160 15463 EventBase.cpp:292] EventBase(): Starting loop.
I0506 15:33:12.448178 15463 AsyncSocket.cpp:1176] AsyncSocket::ioRead() this=0x7f3fbc001740, fd24, events=2, state=2
I0506 15:33:12.448187 15463 AsyncSocket.cpp:1226] AsyncSocket::handleRead() this=0x7f3fbc001740, fd=24, state=2
I0506 15:33:12.448351 15463 Fiber.cpp:112] Stack usage: 5956
I0506 15:33:12.448395 15463 AsyncSocket.cpp:188] new AsyncSocket(0x7f3fbc035250, evb=0x273cc58)
I0506 15:33:12.448460 15463 AsyncSocket.cpp:374] AsyncSocket::connect(this=0x7f3fbc035250, evb=0x273cc58, fd=25, host=10.99.99.1:11211
I0506 15:33:12.448557 15463 Fiber.cpp:112] Stack usage: 6084
I0506 15:33:12.448591 15463 Fiber.cpp:112] Stack usage: 2660
I0506 15:33:12.448604 15463 AsyncSocket.cpp:188] new AsyncSocket(0x7f3fbc035e70, evb=0x273cc58)
I0506 15:33:12.448627 15463 AsyncSocket.cpp:374] AsyncSocket::connect(this=0x7f3fbc035e70, evb=0x273cc58, fd=26, host=10.99.99.2:11211
I0506 15:33:12.448665 15463 Fiber.cpp:112] Stack usage: 2788
I0506 15:33:12.448706 15463 Fiber.cpp:112] Stack usage: 2660
I0506 15:33:12.448725 15463 AsyncSocket.cpp:188] new AsyncSocket(0x7f3fbc0366c0, evb=0x273cc58)
I0506 15:33:12.448748 15463 AsyncSocket.cpp:374] AsyncSocket::connect(this=0x7f3fbc0366c0, evb=0x273cc58, fd=27, host=10.99.99.3:11211
I0506 15:33:12.448786 15463 Fiber.cpp:112] Stack usage: 2788
I0506 15:33:12.448834 15463 Fiber.cpp:112] Stack usage: 2660
I0506 15:33:12.448854 15463 AsyncSocket.cpp:188] new AsyncSocket(0x7f3fbc036f10, evb=0x273cc58)
I0506 15:33:12.448889 15463 AsyncSocket.cpp:374] AsyncSocket::connect(this=0x7f3fbc036f10, evb=0x273cc58, fd=28, host=10.99.99.4:11211
I0506 15:33:12.448938 15463 Fiber.cpp:112] Stack usage: 2788
I0506 15:33:12.448969 15463 Fiber.cpp:112] Stack usage: 2660
I0506 15:33:12.448988 15463 AsyncSocket.cpp:188] new AsyncSocket(0x7f3fbc037760, evb=0x273cc58)
I0506 15:33:12.449019 15463 AsyncSocket.cpp:374] AsyncSocket::connect(this=0x7f3fbc037760, evb=0x273cc58, fd=29, host=10.99.99.5:11211
I0506 15:33:12.449059 15463 Fiber.cpp:112] Stack usage: 2788
I0506 15:33:12.449092 15463 Fiber.cpp:112] Stack usage: 2660
I0506 15:33:12.449108 15463 AsyncSocket.cpp:188] new AsyncSocket(0x7f3fbc037fb0, evb=0x273cc58)
I0506 15:33:12.449167 15463 AsyncSocket.cpp:374] AsyncSocket::connect(this=0x7f3fbc037fb0, evb=0x273cc58, fd=30, host=10.99.99.6:11211
I0506 15:33:12.449211 15463 Fiber.cpp:112] Stack usage: 2788
I0506 15:33:12.449245 15463 Fiber.cpp:112] Stack usage: 2660
I0506 15:33:12.449268 15463 AsyncSocket.cpp:188] new AsyncSocket(0x7f3fbc038800, evb=0x273cc58)
F0506 15:33:12.452230 15463 FiberManager-inl.h:449] Exception St12system_error with message 'Failed to resolve address for "mcrouter07.dev": Name or service not known (error=-2): Unknown error -2' was thrown in FiberManager with context 'running immediateFunc_'
*** Check failure stack trace: ***
@ 0x7f3fd11afdaa (unknown)
@ 0x7f3fd11afce4 (unknown)
@ 0x7f3fd11af6e6 (unknown)
@ 0x7f3fd11af4fb (unknown)
@ 0x46155f _ZZN5folly6fibers12FiberManagerC1IN8facebook8memcache8mcrouter11fiber_local6detail20McrouterFiberContextEEENS0_9LocalTypeIT_EESt10unique_ptrINS0_14LoopControllerESt14default_deleteISD_EENS1_7OptionsEENKUlNSt15__exception_ptr13exception_ptrESsE_clESJ_Ss
@ 0x461642 _ZNSt17_Function_handlerIFvNSt15__exception_ptr13exception_ptrESsEZN5folly6fibers12FiberManagerC1IN8facebook8memcache8mcrouter11fiber_local6detail20McrouterFiberContextEEENS4_9LocalTypeIT_EESt10unique_ptrINS4_14LoopControllerESt14default_deleteISH_EENS5_7OptionsEEUlS1_SsE_E9_M_invokeERKSt9_Any_dataS1_Ss
@ 0x460997 std::function<>::operator()()
@ 0x460dfd folly::fibers::FiberManager::runReadyFiber()
@ 0x461026 folly::fibers::FiberManager::loopUntilNoReady()
@ 0x7f3fd2113094 folly::EventBase::runLoopCallbacks()
@ 0x7f3fd2113332 folly::EventBase::loopBody()
@ 0x43300a _ZNSt17_Function_handlerIFvmRN5folly9EventBaseERN8facebook8memcache19AsyncMcServerWorkerEEZNS4_8mcrouter9runServerERKNS8_25McrouterStandaloneOptionsERNS8_16McrouterInstanceEEUlmS2_S6_E_E9_M_invokeERKSt9_Any_datamS2_S6_
@ 0x5f09ad _ZNSt6thread5_ImplISt12_Bind_simpleIFZN8facebook8memcache14McServerThread5spawnESt8functionIFvmRN5folly9EventBaseERNS3_19AsyncMcServerWorkerEEEmEUlvE0_vEEE6_M_runEv
@ 0x7f3fd0712bf0 (unknown)
@ 0x7f3fd096d182 start_thread
@ 0x7f3fcfe7a38d (unknown)
@ (nil) (unknown)