BUG 生成时候系统报错

[jony4]

BUG 生成时候系统报错

[jony4]

(pid=gcs_server) *** SIGSEGV received at time=1719541509 on cpu 3 ***
*** SIGSEGV received at time=1719541526 on cpu 31 ***
PC: @     0x7356d8e80806  (unknown)  grpc_core::MetadataMap<>::~MetadataMap()
    @     0x735816228420       3376  (unknown)
    @     0x7356d91de24a         48  grpc_core::FilterStackCall::~FilterStackCall()
    @     0x7356d91da4fe         32  grpc_core::Call::DeleteThis()
    @     0x7356d9262374         80  grpc_core::ExecCtx::Flush()
    @     0x7356d8ddf822         32  grpc_core::ExecCtx::~ExecCtx()
    @     0x7356d91d6da7        208  grpc_core::FilterStackCall::ExternalUnref()
    @     0x7356d91d89ab        160  grpc_call_unref
    @     0x7356d8dde010         64  grpc::ClientContext::~ClientContext()
    @     0x7356d88d363a         32  std::_Sp_counted_base<>::_M_release()
    @     0x7356d8b8e147       1376  ray::gcs::PythonGcsSubscriber::DoPoll()
    @     0x7356d8b8eb56        160  ray::gcs::PythonGcsSubscriber::PollLogs()
    @     0x7356d892a0e0        336  __pyx_pw_3ray_7_raylet_16GcsLogSubscriber_3poll()
    @           0x4fdbe2   14269040  method_vectorcall_VARARGS_KEYWORDS
    @           0x740ce0  (unknown)  (unknown)
[2024-06-28 02:25:26,286 E 77 2147] logging.cc:343: *** SIGSEGV received at time=1719541526 on cpu 31 ***
[2024-06-28 02:25:26,286 E 77 2147] logging.cc:343: PC: @     0x7356d8e80806  (unknown)  grpc_core::MetadataMap<>::~MetadataMap()
[2024-06-28 02:25:26,287 E 77 2147] logging.cc:343:     @     0x735816228420       3376  (unknown)
[2024-06-28 02:25:26,287 E 77 2147] logging.cc:343:     @     0x7356d91de24a         48  grpc_core::FilterStackCall::~FilterStackCall()
[2024-06-28 02:25:26,287 E 77 2147] logging.cc:343:     @     0x7356d91da4fe         32  grpc_core::Call::DeleteThis()
[2024-06-28 02:25:26,287 E 77 2147] logging.cc:343:     @     0x7356d9262374         80  grpc_core::ExecCtx::Flush()
[2024-06-28 02:25:26,287 E 77 2147] logging.cc:343:     @     0x7356d8ddf822         32  grpc_core::ExecCtx::~ExecCtx()
[2024-06-28 02:25:26,287 E 77 2147] logging.cc:343:     @     0x7356d91d6da7        208  grpc_core::FilterStackCall::ExternalUnref()
[2024-06-28 02:25:26,287 E 77 2147] logging.cc:343:     @     0x7356d91d89ab        160  grpc_call_unref
[2024-06-28 02:25:26,287 E 77 2147] logging.cc:343:     @     0x7356d8dde010         64  grpc::ClientContext::~ClientContext()
[2024-06-28 02:25:26,287 E 77 2147] logging.cc:343:     @     0x7356d88d363a         32  std::_Sp_counted_base<>::_M_release()
[2024-06-28 02:25:26,287 E 77 2147] logging.cc:343:     @     0x7356d8b8e147       1376  ray::gcs::PythonGcsSubscriber::DoPoll()
[2024-06-28 02:25:26,287 E 77 2147] logging.cc:343:     @     0x7356d8b8eb56        160  ray::gcs::PythonGcsSubscriber::PollLogs()
[2024-06-28 02:25:26,287 E 77 2147] logging.cc:343:     @     0x7356d892a0e0        336  __pyx_pw_3ray_7_raylet_16GcsLogSubscriber_3poll()
[2024-06-28 02:25:26,287 E 77 2147] logging.cc:343:     @           0x4fdbe2   14269040  method_vectorcall_VARARGS_KEYWORDS
[2024-06-28 02:25:26,288 E 77 2147] logging.cc:343:     @           0x740ce0  (unknown)  (unknown)
Fatal Python error: Segmentation fault

Stack (most recent call first):
  File "/opt/conda/lib/python3.10/site-packages/ray/_private/worker.py", line 898 in print_logs
  File "/opt/conda/lib/python3.10/threading.py", line 953 in run
  File "/opt/conda/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/opt/conda/lib/python3.10/threading.py", line 973 in _bootstrap

Extension modules: xoscar.context, xoscar.core, xoscar._utils, mkl._mklinit, mkl._py_mkl_service, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, xoscar.serialization.core, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, xoscar.backends.message, psutil._psutil_linux, psutil._psutil_posix, cython.cimports.libc.math, markupsafe._speedups, _brotli, zstandard.backend_c, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, simplejson._speedups, yaml._yaml, PIL._imaging, regex._regex, sentencepiece._sentencepiece, msgpack._cmsgpack, google._upb._message, setproctitle, uvloop.loop, ray._raylet, pyarrow._json (total: 113)

[ChengjieLi28]

@jony4 看起来是vllm自己的错误导致了segmentation fault，自己尝试升降级vllm，或者去vllm官方repo寻找解决方案，不属于xinference范畴。

[jony4]

使用的 docker 镜像方式部署的。

[jony4]

我升级下最新的 docker 镜像看看