Comments (4)
richardliaw
commented on April 27, 2024
New error:
[FATAL] (photon_client.c:26) Check failure: type == EXECUTE_TASK
[FATAL] (photon_client.c:26) Check failure: type == EXECUTE_TASK
[FATAL] (photon_client.c:26) Check failure: type == EXECUTE_TASK
[FATAL] (photon_client.c:26) Check failure: type == EXECUTE_TASK
[FATAL] (photon_client.c:26) Check failure: type == EXECUTE_TASK
[FATAL] (photon_client.c:26) Check failure: type == EXECUTE_TASK
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/photon/photon/libphoton.so/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/photon/photon/libphoton.so/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/photon/photon/libphoton.so/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/photon/photon/libphoton.so/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/photon/photon/libphoton.so((photon_get_task((+0xphoton_get_taskphoton_get_taskphoton_get_task+0xa8+0x+0xa8a8))a8[0x[0x)7fa22e66b7c8)]
7fefd7c9e7c8[0x[0x7f4470e1d7c8/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/photon/photon/libphoton.so]
7fb4ce6547c8]
(/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/photon/photon/libphoton.so]
+0x(/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/photon/photon/libphoton.so+0x/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/photon/photon/libphoton.so42034203())[0x+0x(7fa22e669203+0x4203)[0x4203]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/photon/photon/libphoton.so(photon_get_task+0xa8)[0x7f11425707c8]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/photon/photon/libphoton.so/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0((+0x4203PyEval_EvalFrameEx+0x)[0x82dc)7f114256e203]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x82dc)[0x7f114d661f3c]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f114d6631ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f114d6621f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f114d6631ce]
(/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0xphoton_get_task+0x32)a8)[0x[0x7f114d6632e27fef088ce7c8]
]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/photon/photon/libphoton.so(+0x4203)[0x7fef088cc203]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.07f4470e1b203]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x82dc))[0x7fef139bff3c]
[0x[0x7fb4ce6522037fefd7c9c203/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0]
]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0[0x)[0x7f114d683960]
(7fa23975cf3cPyEval_EvalCodeEx+0x89e)[0x/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0]
7fef139c11ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x82dc)[0x7fb4d9745f3c]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0((PyEval_EvalFrameEx/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0+0xPyEval_EvalFrameEx82dc()PyEval_EvalFrameEx+0x(+0x8596PyRun_SimpleFileExFlags82dc+0x)/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0ef((PyEval_EvalCodeEx)[0x[0xPyEval_EvalCodeEx+0x+0x89e7fefe2d8ff3c89e))]
7f447bf0ef3c[0x[0x7fb4d97471ce7fa23975e1ce)]
[0x]
7f114d683b3f]
[0x/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0]
7fef139c01f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fb4d97461f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0)[0x7fef139c11ce]
(PyEval_EvalCodeEx+0x89e)([0x/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.07fb4d97471ce((PyEval_EvalCodeExPyEval_EvalCodePy_Main+0x+0x]
3289e+0x))[0x[0x7fef139c12e2(7fefe2d911ce]
]
ca4)[0x7f114d699484]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7fef139e1960]
(PyEval_EvalCode+0x/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0x32)ef)[0x7fef139e1b3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(Py_Main+0xca4/lib/x86_64-linux-gnu/libc.so.6)[0x7fef139f7484]
(__libc_start_main+0xf5)[0x7f114c896f45]
/lib/x86_64-linux-gnu/libc.so.6(python[0x400649__libc_start_main+0xf5)[0x7fef12bf4f45]
]
PyEval_EvalCodeEx+0x89e)[0x[0x7f447bf101ce]
7fb4d97472e2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.08596)[0x7f447bf0f1f6]
(/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0PyRun_FileExFlags+0xb0)[0x(PyEval_EvalFrameEx+0x8596)[0x7fefe2d901f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.07fb4d9767960]
python(PyEval_EvalCodeEx[0x400649]
+0x89e)[0x/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.07f447bf101ce((]
PyEval_EvalCodeExPyRun_SimpleFileExFlags+0x+0x89eef)[0x)[0x7fefe2d911ce7fb4d9767b3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f447bf102e2]
]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0((Py_Main+0xPyRun_FileExFlags+0xb0ca4)[0x/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0)(7fb4d977d484[0xPyEval_EvalCode7f447bf30960+0x]
32)[0x7fefe2d912e2]
]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0()[0x7fefe2db1960PyRun_SimpleFileExFlags+0x]
ef)[0x7f447bf30b3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xef)/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0[0x7fefe2db1b3f]
(Py_Main+0xca4)[0x/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(7f447bf46484]
Py_Main+0xca4)[0x7fefe2dc7484]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f447b143f45]
/lib/x86_64-linux-gnu/libc.so.6(python__libc_start_main+0x[0x400649f5)[0x]
7fefe1fc4f45]
python[0x400649]
[0x/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fb4d897af457fa23975d1f6]
]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(pythonPyEval_EvalCodeEx+0x89e[0x400649)[0x]
7fa23975e1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7fa23975e2e2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7fa23977e960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xef)[0x7fa23977eb3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(Py_Main+0xca4)[0x7fa239794484]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fa238991f45]
python[0x400649]
[INFO] (plasma_manager.c:832) Disconnecting client on fd 25
[INFO] (plasma_manager.c:832) Disconnecting client on fd 21
[INFO] (plasma_manager.c:832) Disconnecting client on fd 18
[INFO] (plasma_manager.c:832) Disconnecting client on fd 20
[INFO] (plasma_manager.c:832) Disconnecting client on fd 29
[INFO] (plasma_manager.c:832) Disconnecting client on fd 24
from ray.
robertnishihara
commented on April 27, 2024
Can you say a bit more about how to reproduce the error (e.g., is there a simple example that reproduces this error)?
from ray.
richardliaw
commented on April 27, 2024
New error:
5e-16 -0.00580101 0.0379549 0.00580101
5.60444 8.86487 110 241.302 2793.38 0.0994867 34 93728 1058 2263.5 2756.71 3311.75 4149 0.933175 2.54257 769.939 4400 900 6.07966e-16 -0.00513754 0.0328048 0.00513754
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 27
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 28
[FATAL] (/home/ubuntu/ray/src/plasma/plasma_client.c:194: errno: Resource temporarily unavailable) Check failure: plasma_receive_reply(conn->store_conn, sizeof(reply), &reply) >= 0
[FATAL] (/home/ubuntu/ray/src/plasma/plasma_client.c:194: errno: Connection reset by peer) Check failure: plasma_receive_reply(conn->store_conn, sizeof(reply), &reply) >= 0
[FATAL] (/home/ubuntu/ray/src/plasma/plasma_client.c:194: errno: Connection reset by peer) Check failure: plasma_receive_reply(conn->store_conn, sizeof(reply), &reply) >= 0
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(plasma_create+0x1b7)[0x7fcf0e008d91]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(PyPlasma_create+0x115)[0x7fcf0e004659]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8942)[0x7fcf193505a2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fcf193511ce]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(plasma_create(PyEval_EvalFrameEx+0x1b7+0x8596))[0x[0x7fcf193501f67f6494084d91]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(PyPlasma_create+0x115)[0x7f6494080659]
]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8942)[0x7f649f3cc5a2]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0)((plasma_create[0x+0xPyEval_EvalCodeEx1b77f649f3cd1ce)+0x]
[0x7f1ec8b38d91/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f649f3cc1f6]
89e]
)[0x7fcf193511ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0PyEval_EvalCodeEx(PyPlasma_create(PyEval_EvalFrameEx+0x+0x+0x89e8596))115[0x[0x7f649f3cd1ce7fcf193501f6]
)/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fcf193511ce]
]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fcf193501f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f649f3cc1f6]
+0x89e)[0x7fcf193511ce[0x]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx7f1ec8b34659]
+0x89e)[0x7f649f3cd1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0)[0x7fcf193501f6(]
PyEval_EvalFrameEx+0x8596)/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0([0x7f649f3cc1f6PyEval_EvalCodeEx+0x89e]
)[0x7fcf193511ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0((PyEval_EvalFrameEx+0xPyEval_EvalCodeEx+0x89ePyEval_EvalFrameEx8942)+0x[0x8596)7f649f3cd1ce)]
[0x7fcf193501f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fcf193511ce]
[0x/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.07f1ed3e805a2(]
(PyEval_EvalCodePyEval_EvalFrameEx+0x+0x32)8596)[0x7fcf193512e2]
[0x7f649f3cc1f6/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeExPyRun_FileExFlags+0x+0x/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0b0(89e)PyEval_EvalCodeEx+0x)89e[0x7f1ed3e811ce)[0x[0x7f649f3cd1ce]
]
7fcf19371960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0+0x8596/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0()PyRun_SimpleFileExFlags[0x7f649f3cc1f6(+0xPyEval_EvalFrameExef+0x8596)]
)[0x7fcf19371b3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(+0x89ePy_Main+0xca4))[0x7fcf19387484]
[0x7f649f3cd1ce]
[0x/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fcf18584f45]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f649f3cd2e2]
7f1ed3e801f6python[0x400649]
]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7f649f3ed960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0PyEval_EvalCodeEx(PyRun_SimpleFileExFlags+0x+0xef)[0x89e7f649f3edb3f]
)[0x7f1ed3e811ce/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0]
(Py_Main+0xca4)[0x7f649f403484]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0/lib/x86_64-linux-gnu/libc.so.6(PyEval_EvalFrameEx(__libc_start_main+0x+0xf5)[0x85967f649e600f45]
)python[0x400649[0x]
7f1ed3e801f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f1ed3e811ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f1ed3e801f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f1ed3e811ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f1ed3e801f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f1ed3e811ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f1ed3e801f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f1ed3e811ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f1ed3e812e2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7f1ed3ea1960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xef)[0x7f1ed3ea1b3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(Py_Main+0xca4)[0x7f1ed3eb7484]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f1ed30b4f45]
python[0x400649]
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 18
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 19
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 23
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 24
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 20
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 21
[FATAL] (/home/ubuntu/ray/src/plasma/plasma_client.c:192: errno: Broken pipe) Check failure: plasma_send_request(conn->store_conn, PLASMA_CREATE, &req) >= 0
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(plasma_create+0x100)[0x7ff45c932cda]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(PyPlasma_create+0x115)[0x7ff45c92e659]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8942)[0x7ff467c7a5a2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7ff467c7b1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7ff467c7a1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7ff467c7b1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7ff467c7a1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7ff467c7b1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7ff467c7a1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7ff467c7b1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7ff467c7a1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7ff467c7b1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7ff467c7a1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7ff467c7b1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7ff467c7b2e2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7ff467c9b960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xef)[0x7ff467c9bb3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(Py_Main+0xca4)[0x7ff467cb1484]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ff466eaef45]
python[0x400649]
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 16
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 17
[FATAL] (/home/ubuntu/ray/src/plasma/plasma_client.c:192: errno: Broken pipe) Check failure: plasma_send_request(conn->store_conn, PLASMA_CREATE, &req) >= 0
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(plasma_create+0x100)[0x7f41c63a6cda]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(PyPlasma_create+0x115)[0x7f41c63a2659]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8942)[0x7f41d16ee5a2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f41d16ef1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f41d16ee1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f41d16ef1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f41d16ee1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f41d16ef1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f41d16ee1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f41d16ef1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f41d16ee1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f41d16ef1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f41d16ee1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f41d16ef1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f41d16ef2e2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7f41d170f960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xef)[0x7f41d170fb3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(Py_Main+0xca4)[0x7f41d1725484]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f41d0922f45]
python[0x400649]
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 21
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 22
[FATAL] (/home/ubuntu/ray/src/plasma/plasma_client.c:192: errno: Broken pipe) Check failure: plasma_send_request(conn->store_conn, PLASMA_CREATE, &req) >= 0
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(plasma_create+0x100)[0x7f406cfc5cda]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(PyPlasma_create+0x115)[0x7f406cfc1659]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8942)[0x7f407830d5a2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f407830e1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f407830d1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f407830e1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f407830d1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f407830e1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f407830d1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f407830e1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f407830d1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f407830e1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f407830d1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f407830e1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f407830e2e2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7f407832e960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xef)[0x7f407832eb3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(Py_Main+0xca4)[0x7f4078344484]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f4077541f45]
python[0x400649]
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 17
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 18
[FATAL] (/home/ubuntu/ray/src/plasma/plasma_client.c:192: errno: Broken pipe) Check failure: plasma_send_request(conn->store_conn, PLASMA_CREATE, &req) >= 0
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(plasma_create+0x100)[0x7fd6a234dcda]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(PyPlasma_create+0x115)[0x7fd6a2349659]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8942)[0x7fd6ad6955a2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fd6ad6961ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fd6ad6951f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fd6ad6961ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fd6ad6951f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fd6ad6961ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fd6ad6951f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fd6ad6961ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fd6ad6951f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fd6ad6961ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fd6ad6951f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fd6ad6961ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7fd6ad6962e2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7fd6ad6b6960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xef)[0x7fd6ad6b6b3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(Py_Main+0xca4)[0x7fd6ad6cc484]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fd6ac8c9f45]
python[0x400649]
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 22
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 23
[FATAL] (/home/ubuntu/ray/src/plasma/plasma_client.c:192: errno: Broken pipe) Check failure: plasma_send_request(conn->store_conn, PLASMA_CREATE, &req) >= 0
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(plasma_create+0x100)[0x7f40ff1d4cda]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(PyPlasma_create+0x115)[0x7f40ff1d0659]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8942)[0x7f410a51c5a2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f410a51d1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f410a51c1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f410a51d1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f410a51c1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f410a51d1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f410a51c1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f410a51d1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f410a51c1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f410a51d1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f410a51c1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f410a51d1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f410a51d2e2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7f410a53d960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xef)[0x7f410a53db3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(Py_Main+0xca4)[0x7f410a553484]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f4109750f45]
python[0x400649]
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 28
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 29
[FATAL] (/home/ubuntu/ray/src/plasma/plasma_client.c:192: errno: Broken pipe) Check failure: plasma_send_request(conn->store_conn, PLASMA_CREATE, &req) >= 0
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(plasma_create+0x100)[0x7f16df1b4cda]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(PyPlasma_create+0x115)[0x7f16df1b0659]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8942)[0x7f16ea4fc5a2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f16ea4fd1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f16ea4fc1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f16ea4fd1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f16ea4fc1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f16ea4fd1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f16ea4fc1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f16ea4fd1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f16ea4fc1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f16ea4fd1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f16ea4fc1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f16ea4fd1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f16ea4fd2e2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7f16ea51d960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xef)[0x7f16ea51db3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(Py_Main+0xca4)[0x7f16ea533484]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f16e9730f45]
python[0x400649]
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 25
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 26
[FATAL] (/home/ubuntu/ray/src/plasma/plasma_client.c:192: errno: Broken pipe) Check failure: plasma_send_request(conn->store_conn, PLASMA_CREATE, &req) >= 0
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(plasma_create+0x100)[0x7fa29e024cda]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(PyPlasma_create+0x115)[0x7fa29e020659]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8942)[0x7fa2a936c5a2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fa2a936d1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fa2a936c1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fa2a936d1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fa2a936c1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fa2a936d1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fa2a936c1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fa2a936d1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fa2a936c1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fa2a936d1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fa2a936c1f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fa2a936d1ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7fa2a936d2e2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7fa2a938d960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xef)[0x7fa2a938db3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(Py_Main+0xca4)[0x7fa2a93a3484]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fa2a85a0f45]
python[0x400649]
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 29
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 30
[FATAL] (/home/ubuntu/ray/src/plasma/plasma_client.c:192: errno: Broken pipe) Check failure: plasma_send_request(conn->store_conn, PLASMA_CREATE, &req) >= 0
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(plasma_create+0x100)[0x7f13163fdcda]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(PyPlasma_create+0x115)[0x7f13163f9659]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8942)[0x7f13217455a2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f13217461ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f13217451f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f13217461ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f13217451f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f13217461ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f13217451f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f13217461ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f13217451f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f13217461ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f13217451f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f13217461ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f13217462e2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7f1321766960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xef)[0x7f1321766b3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(Py_Main+0xca4)[0x7f132177c484]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f1320979f45]
python[0x400649]
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 19
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 20
[FATAL] (/home/ubuntu/ray/src/plasma/plasma_client.c:192: errno: Broken pipe) Check failure: plasma_send_request(conn->store_conn, PLASMA_CREATE, &req) >= 0
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(plasma_create+0x100)[0x7f4fb6a49cda]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(PyPlasma_create+0x115)[0x7f4fb6a45659]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8942)[0x7f4fc1d915a2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f4fc1d921ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f4fc1d911f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f4fc1d921ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f4fc1d911f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f4fc1d921ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f4fc1d911f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f4fc1d921ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f4fc1d911f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f4fc1d921ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7f4fc1d911f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7f4fc1d921ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7f4fc1d922e2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7f4fc1db2960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xef)[0x7f4fc1db2b3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(Py_Main+0xca4)[0x7f4fc1dc8484]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f4fc0fc5f45]
python[0x400649]
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 26
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 27
[FATAL] (/home/ubuntu/ray/src/plasma/plasma_client.c:192: errno: Broken pipe) Check failure: plasma_send_request(conn->store_conn, PLASMA_CREATE, &req) >= 0
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(plasma_create+0x100)[0x7fb4cae89cda]
/home/ubuntu/miniconda2/envs/rllab/lib/python2.7/site-packages/ray-0.0.1-py2.7.egg/plasma/lib/python/libplasma.so(PyPlasma_create+0x115)[0x7fb4cae85659]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8942)[0x7fb4d61d15a2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fb4d61d21ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fb4d61d11f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fb4d61d21ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fb4d61d11f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fb4d61d21ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fb4d61d11f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fb4d61d21ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fb4d61d11f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fb4d61d21ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8596)[0x7fb4d61d11f6]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x89e)[0x7fb4d61d21ce]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x7fb4d61d22e2]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xb0)[0x7fb4d61f2960]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0xef)[0x7fb4d61f2b3f]
/home/ubuntu/miniconda2/envs/rllab/bin/../lib/libpython2.7.so.1.0(Py_Main+0xca4)[0x7fb4d6208484]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fb4d5405f45]
python[0x400649]
[INFO] (photon_scheduler.c:163) Disconnecting client on fd 24
[INFO] (plasma_manager.c:1427) Disconnecting client on fd 25
from ray.
robertnishihara
commented on April 27, 2024
I believe the error was due to running on a machine with more memory than disk. The object store created its memory mapped files in /tmp, which occasionally flushes to disk, but there wasn't enough disk space causing a SIGBUS. This should be fixed by #89. If the error reoccurs, please reopen this issue.
from ray.
Related Issues (20)
- [serve] `InMemoryMetricsStore` leaks memory with handle-side autoscaling metrics enabled
- Phasing out Ray Docker images for Intel-based Mac OSX
- [Serve][High] How to properly configure the load on replicas?
- There are some deprecated items that need to be removed
- CI test linux://rllib:examples/evaluation/evaluation_parallel_to_training_multi_agent_duration_auto_torch_envrunner is flaky HOT 4
- Ray v2.11.0 missing windows distribution HOT 3
- [Ray Tune/ Train] Auth with aws_web_identity_token or use the provided file system provider in runtime config HOT 3
- [<Ray component: Core>] ray raise error on nvidia cuda machine for amdgpu missing
- [Tune] Trials on pre-started game instances
- Release test chaos_torch_batch_inference_16_gpu_300gb_raw.aws failed HOT 3
- [Data] Add `override_num_blocks` parameter to `from_pandas`
- Release test dataset_shuffle_push_based_sort_1tb.aws failed HOT 1
- [Core] Actor/Task cannot be scheduled on worker node. HOT 1
- [Ray core] Stopped job leaks worker HOT 4
- [serve] Support resource-based autoscaling HOT 1
- [Observability / Doc] Add support of ray debugger on windows HOT 5
- [Doc] Python 3.12 `docs` env has conflicts with `..scripts/format.sh` HOT 2
- [Dashboard] `py-spy` profiling initiated from the Ray Dashboard fails if `sudo` is not installed HOT 3
- [Ray Data] map_batches with actors is 25% slower than manually consuming with iter_batches
- [Core][Actors] Duplicate named actor exception should not be lazy if possible HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ray.