baidu/anyq

FAQ-based Question Answering System

anyq's Issues

problem in compiling process.

[ 4%] Built target extern_leveldb
[ 9%] Built target extern_jsoncpp
[ 13%] Built target extern_gtest
[ 18%] Built target extern_xgboost
[ 22%] Built target extern_eigen
[ 23%] Performing build step for 'extern_paddle'
[ 1%] Built target extern_snappy
[ 2%] Built target extern_lib_any
[ 3%] Built target extern_eigen3
[ 4%] Built target extern_warpctc
[ 4%] Built target extern_mklml
[ 4%] Built target extern_gflags
[ 5%] Built target framework_py_proto_init
[ 5%] Built target extern_boost
[ 5%] Built target extern_zlib
[ 5%] Built target profiler_py_proto_init
[ 6%] Performing configure step for 'swig'
[ 7%] Built target extern_threadpool

• test -d Tools/config
• aclocal -I Tools/config
  ./autogen.sh: 11: ./autogen.sh: aclocal: not found
  CMakeFiles/swig.dir/build.make:106: recipe for target 'third_party/swig/src/swig-stamp/swig-configure' failed
  make[5]: *** [third_party/swig/src/swig-stamp/swig-configure] Error 127
  CMakeFiles/Makefile2:370: recipe for target 'CMakeFiles/swig.dir/all' failed
  make[4]: *** [CMakeFiles/swig.dir/all] Error 2
  make[4]: *** Waiting for unfinished jobs....
  [ 7%] Built target extern_snappystream
  [ 8%] Built target extern_glog
  [ 9%] Built target extern_protobuf
  Makefile:105: recipe for target 'all' failed
  make[3]: *** [all] Error 2
  CMakeFiles/extern_paddle.dir/build.make:111: recipe for target 'third_party/paddle/src/extern_paddle-stamp/extern_paddle-build' failed
  make[2]: *** [third_party/paddle/src/extern_paddle-stamp/extern_paddle-build] Error 2
  CMakeFiles/Makefile2:511: recipe for target 'CMakeFiles/extern_paddle.dir/all' failed
  make[1]: *** [CMakeFiles/extern_paddle.dir/all] Error 2
  Makefile:83: recipe for target 'all' failed
  make: *** [all] Error 2

图中的语义索引指的是什么？

SimNet看起来把文本映射了向量语义空间，那么这个语义索引指的什么？是能够帮助快速的进行语义匹配吗？输入一个查询，语义表示把查询映射成向量，语义索引利用这个向量快速查询出相关的doc？这个索引的形式是？

compiling problem

Hi,
I am comping the project but I have got the following error:

[ 4%] Built target extern_leveldb
[ 9%] Built target extern_jsoncpp
[ 13%] Built target extern_gtest
[ 18%] Built target extern_xgboost
[ 22%] Built target extern_eigen
[ 23%] Performing build step for 'extern_paddle'
[ 1%] Built target extern_lib_any
[ 2%] Built target extern_eigen3
[ 2%] Built target extern_gflags
[ 4%] Built target extern_threadpool
[ 4%] Built target extern_snappy
[ 4%] Built target extern_mklml
[ 4%] Built target extern_zlib
[ 5%] Built target extern_boost
[ 6%] Built target profiler_py_proto_init
[ 6%] Built target extern_warpctc
[ 6%] Built target framework_py_proto_init
[ 7%] Built target extern_glog
[ 7%] Built target extern_snappystream
[ 8%] Built target extern_protobuf
[ 8%] Built target stringpiece
[ 9%] Built target header
[ 9%] Built target enforce
[ 9%] Built target cblas
[ 10%] Built target memory_block
Copy generated python proto into directory paddle/fluid/proto.
[ 10%] Built target activation_functions
[ 10%] Generating ../paddle/python/paddle/proto/ParameterService_pb2.py
[ 10%] Generating ../paddle/python/paddle/proto/ParameterServerConfig_pb2.py
[ 10%] Generating ../paddle/python/paddle/proto/TrainerConfig_pb2.py
Copy generated python proto into directory paddle/fluid/proto/profiler.
[ 10%] Generating ../paddle/python/paddle/proto/OptimizerConfig_pb2.py
[ 10%] Generating ../paddle/python/paddle/proto/ParameterConfig_pb2.py
[ 10%] Generating ../paddle/python/paddle/proto/ModelConfig_pb2.py
[ 11%] Generating ../paddle/python/paddle/proto/DataFormat_pb2.py
[ 11%] Generating ../paddle/python/paddle/proto/DataConfig_pb2.py
[ 11%] Built target framework_proto
[ 12%] Built target paddle_proto
[ 12%] Built target chunk
[ 12%] Built target framework_py_proto
[ 12%] Built target ddim
[ 12%] Built target cpu_info
[ 12%] Built target profiler_py_proto
[ 13%] Built target dynamic_loader
[ 13%] Built target gen_proto_py
[ 15%] Built target threadpool
[ 15%] Built target prune
[ 16%] Built target writer
[ 16%] Built target place
[ 17%] Built target profiler_proto
[ 17%] Built target system_allocator
[ 17%] Built target scanner
[ 17%] Built target dynload_warpctc
[ 17%] Built target attribute
[ 17%] Built target scope
[ 18%] Built target buddy_allocator
[ 19%] Built target recordio
[ 19%] Built target var_handle
[ 19%] Built target memcpy
[ 19%] Built target device_tracer
[ 19%] Built target op_proto_maker
[ 19%] Built target op_info
[ 19%] Built target malloc
[ 19%] Built target memory
[ 19%] Built target device_context
[ 19%] Built target vol2col
[ 19%] Built target data_type
[ 20%] Built target sequence2batch
[ 20%] Built target maxouting
[ 20%] Built target sequence_scale
[ 20%] Built target blas
[ 20%] Built target concurrency
[ 20%] Built target pooling
[ 20%] Built target im2col
[ 20%] Built target lstm_compute
[ 20%] Built target sequence_padding
[ 20%] Built target profiler
[ 20%] Built target concat
[ 20%] Built target cos_sim_functor
[ 21%] Built target shape_inference
[ 21%] Built target cross_entropy
[ 22%] Built target unpooling
[ 23%] Built target math_function
[ 23%] Built target tensor
[ 23%] Built target softmax
[ 23%] Built target gru_compute
[ 23%] Built target data_type_transform
[ 23%] Built target data_layout_transform
[ 23%] Built target selected_rows
[ 24%] Built target lod_tensor
[ 24%] Built target context_project
[ 24%] Built target sequence_pooling
[ 25%] Built target data_device_transform
[ 25%] Built target op_handle_base
[ 26%] Built target feed_fetch_method
[ 26%] Built target variable_visitor
[ 26%] Built target reader
[ 26%] Built target lod_rank_table
[ 26%] Built target selected_rows_functor
[ 26%] Built target data_transform
[ 26%] Built target ssa_graph
[ 26%] Built target fetch_op_handle
[ 26%] Built target fuse_vars_op_handle
[ 26%] Built target broadcast_op_handle
[ 26%] Built target reduce_op_handle
[ 26%] Built target all_reduce_op_handle
[ 27%] Built target gather_op_handle
[ 28%] Built target operator
[ 28%] Built target scale_loss_grad_op_handle
[ 29%] Built target ssa_graph_builder
[ 29%] Built target ssa_graph_executor
[ 29%] Built target init
[ 29%] Built target proto_desc
[ 29%] Built target ssa_graph_printer
[ 29%] Built target threaded_ssa_graph_executor
[ 29%] Built target ssa_graph_checker
[ 29%] Built target scope_buffered_ssa_graph_executor
[ 29%] Built target op_registry
[ 29%] Built target scale_op
[ 29%] Built target top_k_op
[ 29%] Built target sigmoid_cross_entropy_with_logits_op
[ 29%] Built target shape_op
[ 29%] Built target sequence_concat_op
[ 30%] Built target executor
[ 30%] Built target smooth_l1_loss_op
[ 31%] Built target computation_op_handle
[ 32%] Built target split_lod_tensor_op
[ 32%] Built target roi_pool_op
[ 33%] Built target rpc_op_handle
[ 33%] Built target uniform_random_op
[ 33%] Built target split_ids_op
[ 33%] Built target split_op
[ 33%] Built target tensor_array_read_write_op
[ 33%] Built target rnn_memory_helper_op
[ 33%] Built target rmsprop_op
[ 33%] Built target reverse_op
[ 33%] Built target select_op
[ 33%] Built target precision_recall_op
[ 34%] Built target reorder_lod_tensor_by_rank_op
[ 34%] Built target reduce_sum_op
[ 34%] Built target norm_op
[ 34%] Built target prelu_op
[ 34%] Built target rank_loss_op
[ 35%] Built target proximal_gd_op
[ 35%] Built target positive_negative_pair_op
[ 35%] Built target split_byref_op
[ 35%] Built target pad_op
[ 35%] Built target sign_op
[ 35%] Built target modified_huber_loss_op
[ 35%] Built target nce_op
[ 35%] Built target minus_op
[ 36%] Built target mean_op
[ 37%] Built target scatter_op
[ 37%] Built target merge_ids_op
[ 37%] Built target mean_iou_op
[ 37%] Built target merge_lod_tensor_op
[ 37%] Built target lod_array_length_op
[ 37%] Built target margin_rank_loss_op
[ 37%] Built target lod_reset_op
[ 37%] Built target log_loss_op
[ 37%] Built target split_selected_rows_op
[ 37%] Built target logical_op
[ 37%] Built target lstm_unit_op
[ 37%] Built target row_conv_op
[ 37%] Built target linear_chain_crf_op
[ 38%] Built target lrn_op
[ 38%] Built target layer_norm_op
[ 38%] Built target spp_op
[ 39%] Built target adamax_op
[ 39%] Built target average_accumulates_op
[ 39%] Built target activation_op
[ 40%] Built target load_combine_op
[ 40%] Built target edit_distance_op
[ 40%] Built target channel_recv_op
[ 41%] Built target channel_send_op
[ 41%] Built target sequence_erase_op
[ 41%] Built target arg_min_op
[ 41%] Built target clip_op
[ 41%] Built target adadelta_op
[ 41%] Built target save_op
[ 41%] Built target save_combine_op
[ 41%] Built target dropout_op
[ 42%] Built target decayed_adagrad_op
[ 42%] Built target detection_map_op
[ 42%] Built target huber_loss_op
[ 42%] Built target fill_constant_batch_size_like_op
[ 42%] Built target while_op
[ 42%] Built target conv_op
[ 42%] Built target warpctc_op
[ 42%] Built target sequence_slice_op
[ 43%] Built target parallel_do_op
[ 43%] Built target elementwise_max_op
[ 43%] Built target softmax_op
[ 43%] Built target batch_norm_op
[ 43%] Built target sequence_conv_op
[ 43%] Built target lookup_sparse_table_op
[ 43%] Built target adam_op
[ 43%] Built target cumsum_op
[ 43%] Built target accuracy_op
[ 43%] Built target cos_sim_op
[ 43%] Built target transpose_op
[ 43%] Built target feed_op
[ 44%] Built target sum_op
[ 44%] Built target reduce_mean_op
[ 44%] Built target max_sequence_len_op
[ 45%] Built target reduce_max_op
[ 45%] Built target sequence_softmax_op
[ 45%] Built target gru_op
[ 45%] Built target l1_norm_op
[ 45%] Built target maxout_op
[ 45%] Built target conv_transpose_op
[ 45%] Built target ctc_align_op
[ 45%] Built target cross_entropy_op
[ 45%] Built target cast_op
[ 45%] Built target auc_op
[ 45%] Built target gaussian_random_batch_size_like_op
[ 45%] Built target mul_op
[ 45%] Built target softmax_with_cross_entropy_op
[ 46%] Built target sequence_pool_op
[ 46%] Built target lookup_table_op
[ 46%] Built target expand_op
[ 46%] Built target recurrent_op
[ 48%] Built target assign_value_op
[ 48%] Built target momentum_op
[ 48%] Built target label_smooth_op
[ 48%] Built target is_empty_op
[ 48%] Built target arg_max_op
[ 48%] Built target print_op
[ 48%] Built target lod_rank_table_op
[ 49%] Built target one_hot_op
[ 49%] Built target concat_op
[ 49%] Built target lstm_op
[ 49%] Built target assign_op
[ 49%] Built target load_op
[ 49%] Built target channel_create_op
[ 49%] Built target sequence_expand_op
[ 49%] Built target pool_op
[ 49%] Built target squared_l2_distance_op
[ 49%] Built target sequence_reshape_op
[ 49%] Built target elementwise_pow_op
[ 49%] Built target sgd_op
[ 49%] Built target fake_dequantize_op
[ 49%] Built target random_crop_op
[ 49%] Built target beam_search_decode_op
[ 49%] Built target elementwise_sub_op
[ 49%] Built target pool_with_index_op
[ 50%] Built target beam_search_op
[ 50%] Built target bilinear_interp_op
[ 50%] Built target slice_op
[ 50%] Built target unpool_op
[ 50%] Built target get_places_op
[ 50%] Built target elementwise_add_op
[ 50%] Built target adagrad_op
[ 50%] Built target shrink_rnn_memory_op
[ 50%] Built target channel_close_op
[ 50%] Built target clip_by_norm_op
[ 50%] Built target fill_zeros_like_op
[ 50%] Built target chunk_eval_op
[ 50%] Built target multiplex_op
[ 50%] Built target compare_op
[ 50%] Built target conditional_block_op
[ 50%] Built target bilinear_tensor_product_op
[ 50%] Built target read_op
[ 50%] Built target lstmp_op
[ 50%] Built target crf_decoding_op
[ 50%] Built target reduce_prod_op
[ 50%] Built target crop_op
[ 50%] Built target gru_unit_op
[ 51%] Built target conv_shift_op
[ 51%] Built target proximal_adagrad_op
[ 51%] Built target increment_op
[ 51%] Built target uniform_random_batch_size_like_op
[ 51%] Built target delete_var_op
[ 51%] Built target fill_constant_op
[ 52%] Built target elementwise_mul_op
[ 52%] Built target gather_op
[ 52%] Built target elementwise_min_op
[ 52%] Built target elementwise_div_op
[ 52%] Built target squared_l2_norm_op
[ 52%] Built target ftrl_op
[ 53%] Built target fill_op
[ 53%] Built target reshape_op
[ 53%] Built target reduce_min_op
[ 53%] Built target gaussian_random_op
[ 53%] Built target fetch_op
[ 53%] Built target hinge_loss_op
[ 53%] Built target im2sequence_op
[ 53%] Built target matmul_op
[ 54%] Built target iou_similarity_op
[ 54%] Built target go_op
[ 54%] Built target target_assign_op
[ 54%] Built target reader_op_registry
[ 54%] Built target mine_hard_examples_op
[ 54%] Built target polygon_box_transform_op
[ 54%] Built target multiclass_nms_op
[ 54%] Built target bipartite_match_op
[ 54%] Built target prior_box_op
[ 54%] Built target box_coder_op
[ 54%] Built target multi_devices_graph_builder
[ 54%] Built target lod_tensor_to_array_op
[ 54%] Built target array_to_lod_tensor_op
[ 54%] Built target create_recordio_file_reader_op
[ 54%] Built target create_shuffle_reader_op
[ 54%] Built target open_files_op
[ 55%] Built target create_threaded_reader_op
[ 55%] Built target create_random_data_generator_op
[ 55%] Built target create_double_buffer_reader_op
[ 55%] Built target create_multi_pass_reader_op
[ 55%] Built target create_batch_reader_op
[ 56%] Built target create_custom_reader_op
[ 56%] Built target ssa_graph_builder_factory
[ 56%] Built target paddle_fluid_api
[ 56%] Built target parallel_executor
make[5]: *** No rule to make target third_party/install/zlib/lib/libz.a', needed by paddle/fluid/inference/libpaddle_fluid.so'. Stop.
make[4]: *** [paddle/fluid/inference/CMakeFiles/paddle_fluid_shared.dir/all] Error 2
make[4]: *** Waiting for unfinished jobs....
[100%] Built target paddle_fluid
make[3]: *** [all] Error 2
make[2]: *** [third_party/paddle/src/extern_paddle-stamp/extern_paddle-build] Error 2
make[1]: *** [CMakeFiles/extern_paddle.dir/all] Error 2
make: *** [all] Error 2

What shall I do to solve it? Thank you.

有没有已经安装所有依赖的国内镜像?

请问有没有已经安装所有依赖的国内镜像?
已经装了4天了,卡在许多要用到git clone的地方.....

如何使用GPU编译？

编译选项增加GPU
'CUDA' is not supported, Please re-compile with WITH_GPU option

请问demo中，你们训练lac词法分析模型使用了多大的数据量？

如题

jaccard similarity issue

When calculate jaccard similarity betwwen query and candidates， why use char set instead of token set？

how --- not in all_model_checkpoint_paths. Manually adding it.

mldl@ub1604:/ub16_prj/AnyQ/tools/simnet/train/tf$ python tf_simnet.py
/usr/local/lib/python2.7/dist-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
2018-08-27 13:11:13.811799: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-08-27 13:11:13.861404: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-08-27 13:11:13.861738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 950M major: 5 minor: 0 memoryClockRate(GHz): 1.124
pciBusID: 0000:01:00.0
totalMemory: 3.95GiB freeMemory: 3.55GiB
2018-08-27 13:11:13.861796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 950M, pci bus id: 0000:01:00.0, compute capability: 5.0)
save model epoch1
INFO:tensorflow:model/pointwise/cnn.epoch1 is not in all_model_checkpoint_paths. Manually adding it.
INFO:tensorflow:model/pointwise/cnn.final is not in all_model_checkpoint_paths. Manually adding it.
mldl@ub1604:/ub16_prj/AnyQ/tools/simnet/train/tf$

compile problem

-- Installing: /home/jcwang/AnyQ/build/third_party/install/paddle/third_party/install/protobuf/lib/cmake/protobuf/protobuf-config-version.cmake
-- Installing: /home/jcwang/AnyQ/build/third_party/install/paddle/third_party/install/protobuf/lib/cmake/protobuf/protobuf-module.cmake
-- Installing: /home/jcwang/AnyQ/build/third_party/install/paddle/third_party/install/protobuf/lib/cmake/protobuf/protobuf-config.cmake
[ 15%] Completed 'extern_protobuf'
[ 15%] Built target extern_protobuf
Makefile:105: recipe for target 'all' failed
make[3]: *** [all] Error 2
CMakeFiles/extern_paddle.dir/build.make:111: recipe for target 'third_party/paddle/src/extern_paddle-stamp/extern_paddle-build' failed
make[2]: *** [third_party/paddle/src/extern_paddle-stamp/extern_paddle-build] Error 2
CMakeFiles/Makefile2:511: recipe for target 'CMakeFiles/extern_paddle.dir/all' failed
make[1]: *** [CMakeFiles/extern_paddle.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

I am a newbie in c++, sorry . I can't solve this problem

编译出错

Scanning dependencies of target memcpy
[ 10%] Building CXX object paddle/fluid/framework/details/CMakeFiles/var_handle.dir/var_handle.cc.o
[ 11%] Building CXX object proto/CMakeFiles/paddle_proto.dir/ParameterConfig.pb.cc.o
[ 11%] Building CXX object paddle/fluid/memory/CMakeFiles/memcpy.dir/memcpy.cc.o
[ 11%] Building CXX object proto/CMakeFiles/paddle_proto.dir/OptimizerConfig.pb.cc.o
[ 11%] Building CXX object proto/CMakeFiles/paddle_proto.dir/ParameterService.pb.cc.o
[ 11%] Building CXX object proto/CMakeFiles/paddle_proto.dir/ParameterServerConfig.pb.cc.o
[ 11%] Built target gen_proto_py
[ 11%] Building CXX object proto/CMakeFiles/paddle_proto.dir/ModelConfig.pb.cc.o
[ 11%] Building CXX object proto/CMakeFiles/paddle_proto.dir/DataConfig.pb.cc.o
[ 11%] Building CXX object proto/CMakeFiles/paddle_proto.dir/DataFormat.pb.cc.o
[ 11%] Building CXX object proto/CMakeFiles/paddle_proto.dir/TrainerConfig.pb.cc.o
In file included from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/printf.h:76:0,
from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/platform/enforce.h:40,
from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/memory/detail/system_allocator.cc:24:
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h: In function ‘const char* paddle::string::tinyformat::detail::streamStateFromFormat(std::ostream&, bool&, int&, const char*, const paddle::string::tinyformat::detail::FormatArg*, int&, int)’:
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:673:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:674:5: note: here
case 'x':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:680:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:681:5: note: here
case 'e':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:686:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:687:5: note: here
case 'f':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:691:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:692:5: note: here
case 'g':
^~~~
In file included from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/printf.h:76:0,
from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/platform/enforce.h:40,
from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/recordio/header.cc:19:
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h: In function ‘const char* paddle::string::tinyformat::detail::streamStateFromFormat(std::ostream&, bool&, int&, const char*, const paddle::string::tinyformat::detail::FormatArg*, int&, int)’:
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:673:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:674:5: note: here
case 'x':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:680:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:681:5: note: here
case 'e':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:686:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:687:5: note: here
case 'f':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:691:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:692:5: note: here
case 'g':
^~~~
In file included from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/printf.h:76:0,
from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/platform/enforce.h:40,
from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/platform/dynload/dynamic_loader.cc:25:
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h: In function ‘const char* paddle::string::tinyformat::detail::streamStateFromFormat(std::ostream&, bool&, int&, const char*, const paddle::string::tinyformat::detail::FormatArg*, int&, int)’:
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:673:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:674:5: note: here
case 'x':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:680:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:681:5: note: here
case 'e':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:686:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:687:5: note: here
case 'f':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:691:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:692:5: note: here
case 'g':
^~~~
cc1plus: all warnings being treated as errors
paddle/fluid/memory/detail/CMakeFiles/system_allocator.dir/build.make:62: recipe for target 'paddle/fluid/memory/detail/CMakeFiles/system_allocator.dir/system_allocator.cc.o' failed
make[5]: *** [paddle/fluid/memory/detail/CMakeFiles/system_allocator.dir/system_allocator.cc.o] Error 1
CMakeFiles/Makefile2:1504: recipe for target 'paddle/fluid/memory/detail/CMakeFiles/system_allocator.dir/all' failed
make[4]: *** [paddle/fluid/memory/detail/CMakeFiles/system_allocator.dir/all] Error 2
make[4]: *** 正在等待未完成的任务....
In file included from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/printf.h:76:0,
from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/platform/enforce.h:40,
from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/framework/threadpool.h:25,
from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/framework/threadpool.cc:15:
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h: In function ‘const char* paddle::string::tinyformat::detail::streamStateFromFormat(std::ostream&, bool&, int&, const char*, const paddle::string::tinyformat::detail::FormatArg*, int&, int)’:
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:673:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:674:5: note: here
case 'x':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:680:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:681:5: note: here
case 'e':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:686:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:687:5: note: here
case 'f':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:691:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:692:5: note: here
case 'g':
^~~~
[ 12%] Linking CXX static library libmemcpy.a
cc1plus: all warnings being treated as errors
paddle/fluid/recordio/CMakeFiles/header.dir/build.make:62: recipe for target 'paddle/fluid/recordio/CMakeFiles/header.dir/header.cc.o' failed
make[5]: *** [paddle/fluid/recordio/CMakeFiles/header.dir/header.cc.o] Error 1
CMakeFiles/Makefile2:22367: recipe for target 'paddle/fluid/recordio/CMakeFiles/header.dir/all' failed
make[4]: *** [paddle/fluid/recordio/CMakeFiles/header.dir/all] Error 2
cc1plus: all warnings being treated as errors
paddle/fluid/platform/dynload/CMakeFiles/dynamic_loader.dir/build.make:62: recipe for target 'paddle/fluid/platform/dynload/CMakeFiles/dynamic_loader.dir/dynamic_loader.cc.o' failed
make[5]: *** [paddle/fluid/platform/dynload/CMakeFiles/dynamic_loader.dir/dynamic_loader.cc.o] Error 1
CMakeFiles/Makefile2:2052: recipe for target 'paddle/fluid/platform/dynload/CMakeFiles/dynamic_loader.dir/all' failed
make[4]: *** [paddle/fluid/platform/dynload/CMakeFiles/dynamic_loader.dir/all] Error 2
[ 12%] Linking CXX static library libvar_handle.a
[ 12%] Built target memcpy
In file included from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/printf.h:76:0,
from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/platform/enforce.h:40,
from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/framework/ddim.h:21,
from /home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/framework/ddim.cc:15:
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h: In function ‘const char* paddle::string::tinyformat::detail::streamStateFromFormat(std::ostream&, bool&, int&, const char*, const paddle::string::tinyformat::detail::FormatArg*, int&, int)’:
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:673:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:674:5: note: here
case 'x':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:680:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:681:5: note: here
case 'e':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:686:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:687:5: note: here
case 'f':
^~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:691:15: error: this statement may fall through [-Werror=implicit-fallthrough=]
out.setf(std::ios::uppercase);
~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/uzhiqiang/AnyQ-master/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/tinyformat.h:692:5: note: here
case 'g':
^~~~
[ 12%] Built target var_handle
cc1plus: all warnings being treated as errors
paddle/fluid/framework/CMakeFiles/threadpool.dir/build.make:62: recipe for target 'paddle/fluid/framework/CMakeFiles/threadpool.dir/threadpool.cc.o' failed
make[5]: *** [paddle/fluid/framework/CMakeFiles/threadpool.dir/threadpool.cc.o] Error 1
CMakeFiles/Makefile2:2726: recipe for target 'paddle/fluid/framework/CMakeFiles/threadpool.dir/all' failed
make[4]: *** [paddle/fluid/framework/CMakeFiles/threadpool.dir/all] Error 2
cc1plus: all warnings being treated as errors
paddle/fluid/framework/CMakeFiles/ddim.dir/build.make:62: recipe for target 'paddle/fluid/framework/CMakeFiles/ddim.dir/ddim.cc.o' failed
make[5]: *** [paddle/fluid/framework/CMakeFiles/ddim.dir/ddim.cc.o] Error 1
CMakeFiles/Makefile2:3263: recipe for target 'paddle/fluid/framework/CMakeFiles/ddim.dir/all' failed
make[4]: *** [paddle/fluid/framework/CMakeFiles/ddim.dir/all] Error 2
[ 12%] Linking CXX static library libframework_proto.a
[ 12%] Built target framework_proto
[ 12%] Linking CXX static library libpaddle_proto.a
[ 12%] Built target paddle_proto
Makefile:105: recipe for target 'all' failed
make[3]: *** [all] Error 2
CMakeFiles/extern_paddle.dir/build.make:111: recipe for target 'third_party/paddle/src/extern_paddle-stamp/extern_paddle-build' failed
make[2]: *** [third_party/paddle/src/extern_paddle-stamp/extern_paddle-build] Error 2
CMakeFiles/Makefile2:857: recipe for target 'CMakeFiles/extern_paddle.dir/all' failed
make[1]: *** [CMakeFiles/extern_paddle.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

系统配置情况
ubuntu 17.04
bison 3.0.5
g++7.2.0

compile error

[100%] Built target paddle_fluid
[100%] Built target paddle_fluid_shared
copying /home/daiwei/AnyQ/build/third_party/paddle/src/extern_paddle/paddle/fluid/inference/.h -> /home/daiwei/AnyQ/build/third_party/install/paddle/fluid_install_dir/paddle/fluid/inference
copying /home/daiwei/AnyQ/build/third_party/install/paddle/paddle/fluid/inference/libpaddle_fluid. -> /home/daiwei/AnyQ/build/third_party/install/paddle/fluid_install_dir/paddle/fluid/inference
[100%] Built target inference_lib
[100%] Built target profiler_py_proto_init
Copy generated python proto into directory paddle/fluid/proto/profiler.
[100%] Built target profiler_py_proto
copying /home/daiwei/AnyQ/build/third_party/paddle/src/extern_paddle/paddle/fluid/platform/.h -> /home/daiwei/AnyQ/build/third_party/install/paddle/fluid_install_dir/paddle/fluid/platform
copying /home/daiwei/AnyQ/build/third_party/paddle/src/extern_paddle/paddle/fluid/platform/dynload/.h -> /home/daiwei/AnyQ/build/third_party/install/paddle/fluid_install_dir/paddle/fluid/platform/dynload
copying /home/daiwei/AnyQ/build/third_party/paddle/src/extern_paddle/paddle/fluid/platform/details/.h -> /home/daiwei/AnyQ/build/third_party/install/paddle/fluid_install_dir/paddle/fluid/platform/details
[100%] Built target platform_lib
copying /home/daiwei/AnyQ/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/.h -> /home/daiwei/AnyQ/build/third_party/install/paddle/fluid_install_dir/paddle/fluid/string
copying /home/daiwei/AnyQ/build/third_party/paddle/src/extern_paddle/paddle/fluid/string/tinyformat/*.h -> /home/daiwei/AnyQ/build/third_party/install/paddle/fluid_install_dir/paddle/fluid/string/tinyformat
[100%] Built target string_lib
[100%] Built target inference_lib_dist
[ 15%] Completed 'extern_paddle'
[ 15%] Built target extern_paddle
make: *** [all] Error 2

My gcc version is 4.8.5 and cmake 3.2.0 Many thanks

运行http服务失败

两个问题：

1. 运行 sh solr_script/anyq_solr.sh solr_script/sample_docs 成功后，再运行 ./run_server
   失败，报错如下：
   E0100 00:00:00.000000 13711 utils.h:90] RAW: load_config_from_file failed, cant open ./example/conf/anyq_brpc.conf
   load_config_from_file failed
   E0100 00:00:00.000000 13711 run_server.cpp:26] RAW: server init failed

其实不是很懂配置文件的位置，由于没看到源代码里面有配置文件，所以我自己在build目录下新建了example/conf/目录，然后把dict.conf, analysis.conf, retrieval.conf,rank.conf这四个配置文件手动放在这个目录下了，在AnyQ的介绍中没有看到anyq_brpc.conf文件

2. 添加语义索引时，在生成语义索引库这一步出错，运行 ./annoy_index_build_tool example/conf/ example/conf/analysis.conf faq/faq_json.index 128 10 semantic.annoy后，报错为：

./annoy_index_build_tool example/conf/ example/conf/analysis.conf faq/faq_json.index 128 10 semantic.annoy
[libprotobuf ERROR /media/sjzwind/5829E2A27A97CAA9/PycharmProjects/AnyQ/build/third_party/protobuf/src/extern_protobuf/src/google/protobuf/text_format.cc:303] Error parsing text-format anyq.DictManagerConfig: Message missing required fields: name
E0100 00:00:00.000000 12080 utils.h:97] RAW: protobuf parse from file[example/conf//dict.conf] failed!
E0100 00:00:00.000000 12080 dict_manager.cpp:43] RAW: load dict.conf from example/conf/ error
E0100 00:00:00.000000 12080 annoy_index_build.cpp:35] RAW: load dict error

还是配置文件的问题，好像是解析出错，对于配置文件确实不太清楚，特来求助。。。

一些问题

simnet里没看到是否支持从word2vec等初始化向量，另外未登录词的是如何处理的？谢谢

训练数据准备问题

Pointwise训练及测试数据格式

训练数据格式：训练数据包含三列，依次为Query1的ID序列（ID间使用空格分割），Query2的ID序列（ID间使用空格分割），Label，每列间使用TAB分割，例如；
1 1 1 1 1 2 2 2 2 2 0
1 1 1 1 1 1 1 1 1 1 1
...

solr clear_document中的问题

运行demo中 sh solr_script/anyq_solr.sh solr_script/sample_docs 命令时，clear_document时报以下错误：
File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python2.7/urllib2.py", line 437, in open
response = meth(req, response)
File "/usr/lib64/python2.7/urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib64/python2.7/urllib2.py", line 475, in error
return self._call_chain(*args)
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib64/python2.7/urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: Server Error
是什么原因？谢谢

启动Solr错误，请求示例失败

请教两个问题，多谢回答~
错误1 ：
xxx@ubuntu1:~/xx/AnyQ/build$ sh solr_script/anyq_solr.sh solr_script/sample_docs
faq-file trans done
nohup: appending output to 'nohup.out'
solr[8900] start success!
[{'indexed': True, 'name': 'question', 'stored': True, 'type': 'text_multi_lang'}, {'indexed': False, 'name': 'answer', 'stored': True, 'type': 'string'}]
*****[Request Error]: Field 'question' already exists.
b'{"responseHeader":{"status":0,"QTime":11}}\n'
b'{"responseHeader":{"status":0,"QTime":160}}\n'
upload file success

错误2：
浏览器中请求示例输入：http://192.168.1.212:8999/anyq?question=注册百度账户时收不到验证码怎么办？显示结果：
[127.0.1.1:8999]Require ServerOptions.session_local_data_factory to be set!
请问是还需要什么配置吗？

paddle.batch好像不会返回不满足batch数量的数据

我用预训练的bow模型在simnet里面做了一下predict，总共的测试数据是947条，batch_size是64，但最后只预测了896条，不知道是不是最后不到64条数据就直接放弃了，还是我的哪里操作出了问题。

c++: internal compiler error 适合内存应该设置多少？

搜了下是编译时内存不足，但内存已经调整成很大了，请问合适的内存大小设置多少？
【环境】
docker CPU:2 Memory: 4.0G Swap: 3.0G
【错误信息】
Scanning dependencies of target sequence_softmax_op
[ 39%] Building CXX object paddle/fluid/operators/CMakeFiles/reduce_max_op.dir/reduce_max_op.cc.o
[ 39%] Linking CXX static library libelementwise_max_op.a
[ 39%] Built target elementwise_max_op
Scanning dependencies of target sequence_pool_op
[ 39%] Building CXX object paddle/fluid/operators/CMakeFiles/sequence_softmax_op.dir/sequence_softmax_op.cc.o
[ 39%] Building CXX object paddle/fluid/operators/CMakeFiles/sequence_pool_op.dir/sequence_pool_op.cc.o
c++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See file:///usr/share/doc/gcc-5/README.Bugs for instructions.
paddle/fluid/operators/CMakeFiles/reduce_min_op.dir/build.make:62: recipe for target 'paddle/fluid/operators/CMakeFiles/reduce_min_op.dir/reduce_min_op.cc.o' failed
make[5]: *** [paddle/fluid/operators/CMakeFiles/reduce_min_op.dir/reduce_min_op.cc.o] Error 4
CMakeFiles/Makefile2:8645: recipe for target 'paddle/fluid/operators/CMakeFiles/reduce_min_op.dir/all' failed
make[4]: *** [paddle/fluid/operators/CMakeFiles/reduce_min_op.dir/all] Error 2
make[4]: *** Waiting for unfinished jobs....
[ 39%] Linking CXX static library libsequence_softmax_op.a
[ 39%] Built target sequence_softmax_op
[ 40%] Linking CXX static library libsequence_pool_op.a
[ 40%] Built target sequence_pool_op
[ 40%] Linking CXX static library libgru_op.a
[ 40%] Built target gru_op
[ 41%] Linking CXX static library libmean_op.a
[ 41%] Built target mean_op
[ 41%] Linking CXX static library libconv_transpose_op.a
[ 41%] Built target conv_transpose_op
[ 41%] Linking CXX static library libadam_op.a
[ 41%] Built target adam_op
[ 42%] Linking CXX static library libsum_op.a
[ 42%] Linking CXX static library libconv_op.a
[ 42%] Built target conv_op
[ 42%] Built target sum_op
[ 42%] Linking CXX static library libnce_op.a
[ 42%] Built target nce_op
[ 43%] Linking CXX static library libparallel_do_op.a
[ 43%] Built target parallel_do_op
[ 43%] Linking CXX static library libreduce_mean_op.a
[ 43%] Built target reduce_mean_op
[ 43%] Linking CXX static library libbatch_norm_op.a
[ 43%] Built target batch_norm_op
[ 43%] Linking CXX static library libreduce_sum_op.a
[ 43%] Built target reduce_sum_op
[ 43%] Linking CXX static library libreduce_max_op.a
[ 43%] Built target reduce_max_op
[ 43%] Linking CXX static library libactivation_op.a
[ 43%] Built target activation_op
Makefile:105: recipe for target 'all' failed
make[3]: *** [all] Error 2
CMakeFiles/extern_paddle.dir/build.make:111: recipe for target 'third_party/paddle/src/extern_paddle-stamp/extern_paddle-build' failed
make[2]: *** [third_party/paddle/src/extern_paddle-stamp/extern_paddle-build] Error 2
CMakeFiles/Makefile2:511: recipe for target 'CMakeFiles/extern_paddle.dir/all' failed
make[1]: *** [CMakeFiles/extern_paddle.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

生成语义索引库能否热加载?

请问生成语义索引库能否热加载? 打算做增量索引。

我尝试将新增的数据根据使用范例

python solr_script/make_json.py solr_script/sample_docs faq/schema_format faq/faq_json
awk -F "\t" '{print ++ind"\t"$0}' faq/faq_json > faq/faq_json.index
./annoy_index_build_tool example/conf/ example/conf/analysis.conf faq/faq_json.index 128 10 semantic.annoy 1>std 2>err
\cp -rf faq/faq_json.index semantic.annoy example/conf

生成faq_json.index与semantic.annoy文件后,直接覆盖。
发现新的索引并没有生效，是不是我使用的方式错误?

Problem of using semantic retrieval

I am experimenting the semantic retrieval according to the tutorial, while got the corruption when starting run_server as follows:

E0100 00:00:00.000000  9225 dual_dict_wrapper.cpp:38] RAW: can't find dict_type:String2RecallItemAdapter
I0100 00:00:00.000000  9225 dual_dict_wrapper.cpp:64] RAW: load dict file:./example/conf/./faq_json.index
I0100 00:00:00.000000  9225 dual_dict_wrapper.cpp:65] RAW: dict file state: Wed Jul 11 10:25:55 2018


Program received signal SIGSEGV, Segmentation fault.
0x00000000007c6cd8 in anyq::DualDictWrapper::reload (this=0x5021ff0) at /root/AnyQ/src/dict/dual_dict_wrapper.cpp:68
68	        if (tmp_dict->release() != 0) {

The error happens due to the plugin of String2RecallItemAdapter has not been registered. However, there does not exist such definition within the project, where could I find it within the project?

运行run_server报错“error while loading shared libraries: libpaddle_fluid.so: cannot open shared object file: No such file or directory”

1. 环境： centos+Python 2.7.5
2. ./run_server报错

error while loading shared libraries: libpaddle_fluid.so: cannot open shared object file: No such file or directory

请教各路大神，如何解决这个问题

插件语言可否不是C++?

您好，想问下如果自定义插件加入AnyQ，可否不是C++语言编写的，比如JAVA，python是否支持？

在训练模型时，是否使用了预训练的词嵌入？

如题

索引用solr是因为开源原因吗？

百度内部的索引系统应该更完善吧？

./autogen.sh: 11: ./autogen.sh: aclocal: not found

在编译的时候，我遇到了下面的错误：
[ 0%] Built target extern_gflags
[ 0%] Built target extern_mklml
[ 1%] Built target extern_threadpool
[ 2%] Performing configure step for 'swig'
[ 3%] Built target extern_snappy
[ 4%] Built target extern_eigen3
[ 4%] Built target extern_zlib

• test -d Tools/config
• aclocal -I Tools/config
  ./autogen.sh: 11: ./autogen.sh: aclocal: not found
  [ 5%] Built target extern_warpctc
  CMakeFiles/swig.dir/build.make:106: recipe for target 'third_party/swig/src/swig-stamp/swig-configure' failed
  make[5]: *** [third_party/swig/src/swig-stamp/swig-configure] Error 127
  CMakeFiles/Makefile2:370: recipe for target 'CMakeFiles/swig.dir/all' failed
  make[4]: *** [CMakeFiles/swig.dir/all] Error 2
  make[4]: *** Waiting for unfinished jobs....
  [ 6%] Built target extern_lib_any
  [ 7%] Built target extern_boost
  [ 7%] Built target profiler_py_proto_init
  [ 7%] Built target framework_py_proto_init
  [ 8%] Built target extern_glog
  [ 9%] Built target extern_protobuf
  [ 9%] Built target extern_snappystream
  Makefile:105: recipe for target 'all' failed
  make[3]: *** [all] Error 2
  CMakeFiles/extern_paddle.dir/build.make:111: recipe for target 'third_party/paddle/src/extern_paddle-stamp/extern_paddle-build' failed
  make[2]: *** [third_party/paddle/src/extern_paddle-stamp/extern_paddle-build] Error 2
  CMakeFiles/Makefile2:511: recipe for target 'CMakeFiles/extern_paddle.dir/all' failed
  make[1]: *** [CMakeFiles/extern_paddle.dir/all] Error 2
  Makefile:83: recipe for target 'all' failed
  make: *** [all] Error 2

protobuf版本问题导致的不能编译

编译 log 如下：
[ 4%] Built target extern_leveldb
[ 9%] Built target extern_jsoncpp
[ 13%] Built target extern_gtest
[ 18%] Built target extern_xgboost
[ 22%] Built target extern_eigen
[ 26%] Built target extern_paddle
[ 31%] Built target extern_curl
[ 35%] Built target extern_lac
[ 39%] Built target extern_boost
[ 43%] Built target extern_zlib
[ 47%] Built target extern_protobuf
[ 52%] Built target extern_gflags
[ 57%] Built target extern_openssl
[ 61%] Built target extern_glog
[ 61%] Performing install step for 'extern_brpc'
Compiling src/mcpack2pb/generator.o
In file included from src/mcpack2pb/generator.cpp:27:0:
./src/idl_options.pb.h:12:2: 错误：#error This file was generated by a newer version of protoc which is
#error This file was generated by a newer version of protoc which is
^
./src/idl_options.pb.h:13:2: 错误：#error incompatible with your Protocol Buffer headers. Please update
#error incompatible with your Protocol Buffer headers. Please update
^
./src/idl_options.pb.h:14:2: 错误：#error your headers.
#error your headers.
^
In file included from ./src/idl_options.pb.h:25:0,
from src/mcpack2pb/generator.cpp:27:
/usr/local/include/google/protobuf/generated_message_table_driven.h: 在函数‘bool google::protobuf::internal::ParseMap(google::protobuf::io::CodedInputStream*, void*)’中:
/usr/local/include/google/protobuf/generated_message_table_driven.h:185:20: 错误：expected nested-name-specifier before ‘MapEntryToMapField’
typedef typename MapEntryToMapField::MapFieldType MapFieldType;
^
/usr/local/include/google/protobuf/generated_message_table_driven.h:185:38: 错误：expected initializer before ‘<’ token
typedef typename MapEntryToMapField::MapFieldType MapFieldType;
^
/usr/local/include/google/protobuf/generated_message_table_driven.h:189:43: 错误：‘MapFieldType’在此作用域中尚未声明
typedef typename Entry::template Parser<MapFieldType, MapType> ParserType;
^
/usr/local/include/google/protobuf/generated_message_table_driven.h:191:33: 错误：expected type-specifier before ‘MapFieldType’
ParserType parser(static_cast<MapFieldType*>(map_field));
^
/usr/local/include/google/protobuf/generated_message_table_driven.h:191:33: 错误：expected ‘>’ before ‘MapFieldType’
/usr/local/include/google/protobuf/generated_message_table_driven.h:191:33: 错误：expected ‘(’ before ‘MapFieldType’
/usr/local/include/google/protobuf/generated_message_table_driven.h:191:46: 错误：expected primary-expression before ‘>’ token
ParserType parser(static_cast<MapFieldType*>(map_field));
^
In file included from ./src/idl_options.pb.h:27:0,
from src/mcpack2pb/generator.cpp:27:
/usr/local/include/google/protobuf/inlined_string_field.h: 在全局域：
/usr/local/include/google/protobuf/inlined_string_field.h:56:22: 错误：expected ‘;’ at end of member declaration
InlinedStringField()
^
/usr/local/include/google/protobuf/inlined_string_field.h:57:5: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE;
^
/usr/local/include/google/protobuf/inlined_string_field.h:61:56: 错误：expected ‘;’ at end of member declaration
const InlinedStringField& from)
^
/usr/local/include/google/protobuf/inlined_string_field.h:62:5: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE;
^
/usr/local/include/google/protobuf/inlined_string_field.h:64:69: 错误：expected ‘;’ at end of member declaration
void ClearToEmpty(const ::std::string* default_value, Arena* arena)
^
/usr/local/include/google/protobuf/inlined_string_field.h:65:7: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:68:31: 错误：expected ‘;’ at end of member declaration
void ClearNonDefaultToEmpty() GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:68:33: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
void ClearNonDefaultToEmpty() GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:71:62: 错误：expected ‘;’ at end of member declaration
void ClearToEmptyNoArena(const ::std::string* default_value)
^
/usr/local/include/google/protobuf/inlined_string_field.h:72:7: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:75:38: 错误：expected ‘;’ at end of member declaration
void ClearNonDefaultToEmptyNoArena()
^
/usr/local/include/google/protobuf/inlined_string_field.h:76:5: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE;
^
/usr/local/include/google/protobuf/inlined_string_field.h:78:71: 错误：expected ‘;’ at end of member declaration
void ClearToDefault(const ::std::string* default_value, Arena* arena)
^
/usr/local/include/google/protobuf/inlined_string_field.h:79:7: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:82:64: 错误：expected ‘;’ at end of member declaration
void ClearToDefaultNoArena(const ::std::string* default_value)
^
/usr/local/include/google/protobuf/inlined_string_field.h:83:5: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE;
^
/usr/local/include/google/protobuf/inlined_string_field.h:85:64: 错误：expected ‘;’ at end of member declaration
void Destroy(const ::std::string* default_value, Arena* arena)
^
/usr/local/include/google/protobuf/inlined_string_field.h:86:7: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:89:57: 错误：expected ‘;’ at end of member declaration
void DestroyNoArena(const ::std::string* default_value)
^
/usr/local/include/google/protobuf/inlined_string_field.h:90:5: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE;
^
/usr/local/include/google/protobuf/inlined_string_field.h:92:30: 错误：expected ‘;’ at end of member declaration
const ::std::string& Get() const GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:92:36: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
const ::std::string& Get() const GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:95:37: 错误：expected ‘;’ at end of member declaration
const ::std::string& GetNoArena() const GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE;
^
/usr/local/include/google/protobuf/inlined_string_field.h:95:43: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
const ::std::string& GetNoArena() const GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE;
^
/usr/local/include/google/protobuf/inlined_string_field.h:97:74: 错误：expected ‘;’ at end of member declaration
::std::string* Mutable(const ::std::string* default_value, Arena* arena)
^
/usr/local/include/google/protobuf/inlined_string_field.h:98:7: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:101:67: 错误：expected ‘;’ at end of member declaration
::std::string* MutableNoArena(const ::std::string* default_value)
^
/usr/local/include/google/protobuf/inlined_string_field.h:102:5: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE;
^
/usr/local/include/google/protobuf/inlined_string_field.h:117:24: 错误：expected ‘;’ at end of member declaration
Arena* arena) GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:117:26: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
Arena* arena) GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:122:28: 错误：expected ‘;’ at end of member declaration
Arena* arena) GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:122:30: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
Arena* arena) GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:126:36: 错误：expected ‘;’ at end of member declaration
StringPiece value) GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE;
^
/usr/local/include/google/protobuf/inlined_string_field.h:126:38: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
StringPiece value) GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE;
^
/usr/local/include/google/protobuf/inlined_string_field.h:130:24: 错误：expected ‘;’ at end of member declaration
Arena* arena) GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:130:26: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
Arena* arena) GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:135:28: 错误：expected ‘;’ at end of member declaration
Arena* arena) GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:135:30: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
Arena* arena) GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE {
^
/usr/local/include/google/protobuf/inlined_string_field.h:139:45: 错误：expected ‘;’ at end of member declaration
const ::std::string& value)
^
/usr/local/include/google/protobuf/inlined_string_field.h:140:5: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE;
^
/usr/local/include/google/protobuf/inlined_string_field.h:154:37: 错误：expected ‘;’ at end of member declaration
void Swap(InlinedStringField* from)
^
/usr/local/include/google/protobuf/inlined_string_field.h:155:5: 错误：‘GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE’不是一个类型名
GOOGLE_PROTOBUF_ATTRIBUTE_ALWAYS_INLINE;
^
In file included from src/mcpack2pb/generator.cpp:27:0:
./src/idl_options.pb.h:42:16: 错误：‘FieldMetadata’不是命名空间‘google::protobuf::internal’中的一个类型名
static const ::google::protobuf::internal::FieldMetadata field_metadata[];
^
./src/idl_options.pb.h:43:16: 错误：‘SerializationTable’不是命名空间‘google::protobuf::internal’中的一个类型名
static const ::google::protobuf::internal::SerializationTable serialization_table[];
^
make[3]: *** [src/mcpack2pb/generator.o] 错误 1
make[2]: *** [third_party/brpc/src/extern_brpc-stamp/extern_brpc-install] 错误 2
make[1]: *** [CMakeFiles/extern_brpc.dir/all] 错误 2
make: *** [all] 错误 2

./run_server的处理过程

两个问题：

1. 可以通过什么方式看到run_server的处理过程的源代码吗，这样对于修改会方便些。
   例如，用curl返回的是乱码，我想在源代码中修改编码方式，不知道怎么操作

2. 我在sample_doc中增加了几个问答对，然后运行 sh solr_script/anyq_solr.sh solr_script/sample_docs来上传数据，这样每次增加了新的问答对都要把所有的数据重新上传一遍吗，还是说可以新建另外的sample_doc，放新的数据来单独上传

感谢解答～～

macOS compile error

[ 9%] Built target framework_py_proto_init
paddle/fluid/framework/CMakeFiles/framework_py_proto.dir/build.make:60: *** target pattern contains no `%'. Stop.
make[4]: *** [paddle/fluid/framework/CMakeFiles/framework_py_proto.dir/all] Error 2
make[4]: *** Waiting for unfinished jobs....
[ 9%] Built target profiler_py_proto_init
[ 11%] Built target extern_glog
[ 12%] Built target extern_snappystream
[ 13%] Built target extern_protobuf
make[3]: *** [all] Error 2
make[2]: *** [third_party/paddle/src/extern_paddle-stamp/extern_paddle-build] Error 2
make[1]: *** [CMakeFiles/extern_paddle.dir/all] Error 2

What's wrong ?

Also I can't get "docker run paddlepaddle/paddle:latest-dev" successful, no process is up!

训练SimNet时报错

INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.DataLossError'>, corrupted record at 0
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](TFRecordReaderV2, input_producer)]]
INFO:tensorflow:model/pointwise/lstm.final is not in all_model_checkpoint_paths. Manually adding it.
Traceback (most recent call last):
File "tf_simnet.py", line 135, in
train(config)
File "tf_simnet.py", line 84, in train
controler.run_trainer(loss, optimizer, conf_dict)
File "/root/workspace/zlong/nlp/anyq/AnyQ/tools/simnet/train/tf/utils/controler.py", line 115, in run_trainer
coord.join(read_thread)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/queue_runner_impl.py", line 252, in _run
enqueue_callable()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1249, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1420, in _call_tf_sessionrun
status, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 516, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.DataLossError: corrupted record at 0
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/device:CPU:0"](TFRecordReaderV2, input_producer)]]

demo启动solr的命令运行失败

sh solr_script/anyq_solr.sh solr_script/sample_docs

测试solr启动没有成功，命令为：

 curl http://localhost:8900/solr/collection1/schema/fields

输出为：

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/collection1/schema/fields. Reason:
<pre>    Not Found</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                
<br/>                                                

</body>
</html>

运行demo不成功

重新在docker上试了一下，编译通过了，./run_server后是：
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:26] RAW: config_name: rank_weights
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:27] RAW: config_type: String2FloatAdapter
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:28] RAW: config_path: ./rank_weights
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:50] RAW: dual dict init: rank_weights, reload:false
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:64] RAW: load dict file:./example/conf/./rank_weights
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:65] RAW: dict file state: Mon Jul 2 08:40:03 2018

I0100 00:00:00.000000 36 dict_adapter.h:62] RAW: ./example/conf/./rank_weights
I0100 00:00:00.000000 36 utils.cpp:67] RAW: hash load ./example/conf/./rank_weights done
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:77] RAW: load dict ./example/conf/./rank_weights success
I0100 00:00:00.000000 36 dict_manager.cpp:59] RAW: dict load success: rank_weights
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:26] RAW: config_name: lac
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:27] RAW: config_type: WordsegAdapter
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:28] RAW: config_path: ./wordseg_utf8
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:50] RAW: dual dict init: lac, reload:false
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:64] RAW: load dict file:./example/conf/./wordseg_utf8
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:65] RAW: dict file state: Mon Jul 2 07:25:18 2018

Loaded q2b dic -- num = 172
Loaded strong punc -- num = 5
Loaded word dic -- num(with oov) = 20940
Loaded tag dic -- num = 57
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0805 05:12:33.740880 36 init.cc:85] 'CUDA' is not supported, Please re-compile with WITH_GPU option
W0805 05:12:33.740921 36 init.cc:101] 'CUDA' is not supported, Please re-compile with WITH_GPU option
Loaded customization dic -- num = 0
I0100 00:00:00.000000 36 wordseg_adapter.cpp:36] RAW: wordseg dict load success.
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:77] RAW: load dict ./example/conf/./wordseg_utf8 success
I0100 00:00:00.000000 36 dict_manager.cpp:59] RAW: dict load success: lac
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:26] RAW: config_name: fluid_simnet
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:27] RAW: config_type: PaddleSimAdapter
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:28] RAW: config_path: ./simnet
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:50] RAW: dual dict init: fluid_simnet, reload:false
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:64] RAW: load dict file:./example/conf/./simnet
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:65] RAW: dict file state: Mon Jul 2 12:32:00 2018

I0100 00:00:00.000000 36 utils.cpp:88] RAW: hash load ./example/conf/./simnet/term2id.dict done
W0805 05:12:34.275743 36 init.cc:85] 'CUDA' is not supported, Please re-compile with WITH_GPU option
W0805 05:12:34.275768 36 init.cc:101] 'CUDA' is not supported, Please re-compile with WITH_GPU option
W0805 05:12:34.275797 36 init.cc:85] 'CUDA' is not supported, Please re-compile with WITH_GPU option
W0805 05:12:34.275807 36 init.cc:101] 'CUDA' is not supported, Please re-compile with WITH_GPU option
I0100 00:00:00.000000 36 paddle_sim_adapter.cpp:66] RAW: paddle model fluid_simnet load success
I0100 00:00:00.000000 36 dual_dict_wrapper.cpp:77] RAW: load dict ./example/conf/./simnet success
I0100 00:00:00.000000 36 dict_manager.cpp:59] RAW: dict load success: fluid_simnet
I0100 00:00:00.000000 36 http_service_impl.cpp:49] RAW: create req preproc plugin default_preproc success
I0100 00:00:00.000000 36 http_service_impl.cpp:63] RAW: create req postproc plugin default_postproc success
I0805 05:12:34.436502 36 server.cpp:975] Server[anyq::HttpServiceImpl] is serving on port=8999.
I0805 05:12:34.437322 36 server.cpp:978] Check out http://9da3a39bb0ef:8999 in web browser.
随后，在浏览器中试了一下：
本地ip:8999/anyq?question=需要使用什么账号登录
得到结果：

而后，在docker试了一下（ctrl+p+q退出了容器）：
curl "127.0.0.1:8999/anyq?question=需要使用什么账号登录"
结果还是：curl: (7) Failed to connect to 127.0.0.1 port 8999: Connection refused

语义召回的疑问

是否语义检索提供的召回和倒排索引提供的召回合并到一起进行匹配和排序？

extern_paddle compile error

[编译环境]
Distributor ID: Debian
Description: Debian GNU/Linux 8.10 (jessie)
Release: 8.10
Codename: jessie
[报错信息]
Makefile:105: recipe for target 'all' failed
make[3]: *** [all] Error 2
CMakeFiles/extern_paddle.dir/build.make:111: recipe for target 'third_party/paddle/src/extern_paddle-stamp/extern_paddle-build' failed
make[2]: *** [third_party/paddle/src/extern_paddle-stamp/extern_paddle-build] Error 2
CMakeFiles/Makefile2:590: recipe for target 'CMakeFiles/extern_paddle.dir/all' failed
make[1]: *** [CMakeFiles/extern_paddle.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

simnet 有没有训练数据集？

simnet 有没有训练数据集？或者其他公开的数据集

如何使用PredictXGBoostModel?

请问一下 我在example/conf/rank.conf 中把

rank_predictor {
type: "PredictLinearModel"
using_dict_name: "rank_weights"
}

替换成下面两种
rank_predictor {
type: "PredictXGBoostModel"
using_dict_name: "rank_weights"
}

rank_predictor {
type: "PredictXGBoostModel"
}

但重启后接收请求，发现都会出现以下错误
Require ServerOptions.session_local_data_factory to be set!

请问该如何配置PredictXGBoostModel呢?

运行run_train.sh报段错误

昨天安装好了AnyQ，在AnyQ/tools/simnet/train/tf/目录下运行sh run_train.sh脚本，报段错误（吐核），我断点试了下，是在utils/controler.py:94(c, _= sess.run([loss, optimizer]))报的。是我操作有问题还是TensorFlow版本问题？

Python版本：Python 2.7.13
TensorFlow版本：TensorFlow-CPU 1.8.0

编译过程中经常因为clone依赖项目导致编译中断和失败

有没有快速无障碍的编译方案 ，求解。

extern_paddle编译报错

生成语义索引库core dumped

生成语义索引库：

./annoy_index_build_tool example/conf/ example/conf/analysis.conf faq/faq_json.index 128 10 semantic.annoy
跑这句时候出现core dumped 具体信息在下面

./annoy_index_build_tool example/conf/ example/conf/analysis.conf faq/faq_json.index 128 10 semantic.annoy 1>std 2>err
Segmentation fault (core dumped)
[daiwei@DL2 build]$ ./annoy_index_build_tool example/conf/ example/conf/analysis.conf faq/faq_json.index 128 10 semantic.annoy
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:26] RAW: config_name: rank_weights
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:27] RAW: config_type: String2FloatAdapter
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:28] RAW: config_path: ./rank_weights
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:50] RAW: dual dict init: rank_weights, reload:false
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:64] RAW: load dict file:example/conf/./rank_weights
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:65] RAW: dict file state: Mon Jul 2 16:40:03 2018

I0100 00:00:00.000000 24511 dict_adapter.h:62] RAW: example/conf/./rank_weights
I0100 00:00:00.000000 24511 utils.cpp:67] RAW: hash load example/conf/./rank_weights done
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:77] RAW: load dict example/conf/./rank_weights success
I0100 00:00:00.000000 24511 dict_manager.cpp:59] RAW: dict load success: rank_weights
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:26] RAW: config_name: lac
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:27] RAW: config_type: WordsegAdapter
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:28] RAW: config_path: ./wordseg_utf8
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:50] RAW: dual dict init: lac, reload:false
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:64] RAW: load dict file:example/conf/./wordseg_utf8
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:65] RAW: dict file state: Mon Jul 2 15:25:18 2018

Loaded q2b dic -- num = 172
Loaded strong punc -- num = 5
Loaded word dic -- num(with oov) = 20940
Loaded tag dic -- num = 57
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0813 14:28:51.788597 24511 init.cc:85] 'CUDA' is not supported, Please re-compile with WITH_GPU option
W0813 14:28:51.788627 24511 init.cc:101] 'CUDA' is not supported, Please re-compile with WITH_GPU option
Loaded customization dic -- num = 0
I0100 00:00:00.000000 24511 wordseg_adapter.cpp:36] RAW: wordseg dict load success.
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:77] RAW: load dict example/conf/./wordseg_utf8 success
I0100 00:00:00.000000 24511 dict_manager.cpp:59] RAW: dict load success: lac
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:26] RAW: config_name: fluid_simnet
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:27] RAW: config_type: TFModelAdapter
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:28] RAW: config_path: ./simnet
E0100 00:00:00.000000 24511 plugin_factory.cpp:28] RAW: create plugin[TFModelAdapter] failed.
E0100 00:00:00.000000 24511 plugin_factory.cpp:28] RAW: create plugin[TFModelAdapter] failed.
E0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:38] RAW: can't find dict_type:TFModelAdapter
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:64] RAW: load dict file:example/conf/./simnet
I0100 00:00:00.000000 24511 dual_dict_wrapper.cpp:65] RAW: dict file state: Mon Jul 2 20:32:00 2018

Segmentation fault (core dumped)

请问这个是因为tfmodel failed么，编译选的是ON， 另外这个tool文件是什么呢？ 谢谢

simnet-tensorflow命令有误

生成数据的命令输入输出地址一样，会导致文件被清空。

编译错误-gflags、glog

编译时，直接make，gflags会出错，改成sudo make就能通过，不清楚这是什么坑；
sudo make后，glog有问题，错误提示如下：
/data/denniszheng/AnyQ/build/third_party/glog/src/glog-0.3.5/src/googletest.h:93: undefined reference to google::FlagRegisterer::FlagRegisterer<std::string>(char const*, char cons t*, char const*, std::string*, std::string*)' 83975 /data/denniszheng/AnyQ/build/third_party/glog/src/glog-0.3.5/src/googletest.h:94: undefined reference to google::FlagRegisterer::FlagRegistererstd::string(char const*, char cons t*, char const*, std::string*, std::string*)'
83976 /data/denniszheng/AnyQ/build/third_party/glog/src/glog-0.3.5/src/googletest.h:96: undefined reference to google::FlagRegisterer::FlagRegisterer<bool>(char const*, char const*, cha r const*, bool*, bool*)' 83977 /data/denniszheng/AnyQ/build/third_party/glog/src/glog-0.3.5/src/googletest.h:100: undefined reference to google::FlagRegisterer::FlagRegisterer(char const*, char const*, cha r const*, int*, int*)'
83978 collect2: error: ld returned 1 exit status
83979 make[3]: *** [logging_unittest] 错误 1
83980 make[2]: *** [third_party/glog/src/extern_glog-stamp/extern_glog-install] 错误 2
83981 make[1]: *** [CMakeFiles/extern_glog.dir/all] 错误 2
83982 make: *** [all] 错误 2

solr灌库失败

1. 目标：在不重启solr的情况下，远程更新FAQ set，希望在本地对FAQ set文件做灵活的增删操作。
2. 操作1: delete engine（非SolrCloud模式）：
   • cmd: curl -X GET -H "Content-Type: application/json" "http://10.11.16.5:8900/solr/admin/cores?wt=json&action=UNLOAD&core=collection1"
   • return: {"responseHeader":{"status":0,"QTime":35}}
3. 操作2：add engine失败后重新把collection1重新load回去
4. clear_doc、upload_doc失败，报错信息如下：
   Traceback (most recent call last): File "solr_script/solr_api.py", line 24, in <module> solr_tools.set_engine_schema(sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5]) File "/data/home/fanzhengfeng/project/AnyQ/demo/solr_script/solr_tools.py", line 117, in set_engine_schema _get_error_message(err.read()) File "/data/home/fanzhengfeng/project/AnyQ/demo/solr_script/solr_tools.py", line 53, in _get_error_message respond_dict = json.loads(respond_str.strip()) File "/data/home/fanzhengfeng/anaconda3/envs/python2/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/data/home/fanzhengfeng/anaconda3/envs/python2/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/data/home/fanzhengfeng/anaconda3/envs/python2/lib/python2.7/json/decoder.py", line 382, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded

对solr不熟悉，不知道是哪里出了问题，求帮助

测试问题

./run_server之后，在浏览器中测试，结果有下面的问题：
I0909 10:45:00.390359 44 method_wordseg.cpp:39] RAW: init wordseg success
I0909 10:45:00.390359 44 analysis_strategy.cpp:74] RAW: create method AnalysisWordseg success
I0909 10:45:00.390359 44 term_retrieval.cpp:77] RAW: create solr q builder equal_solr_q_1 success
I0909 10:45:00.390359 44 retrieval_strategy.cpp:56] RAW: create retrieval plugin term_recall_1 success
I0909 10:45:00.390359 44 wordseg_proc.cpp:33] RAW: init wordseg_feature success
I0909 10:45:00.390359 44 rank_strategy.cpp:100] RAW: create feature wordseg_process success
I0909 10:45:00.390359 44 simnet_paddle_sim.cpp:35] RAW: _output_num:1
I0909 10:45:00.390359 44 rank_strategy.cpp:100] RAW: create feature fluid_simnet_feature success
I0909 10:45:00.390359 44 rank_strategy.cpp:100] RAW: create feature jaccard_sim success
I0909 10:45:00.390359 44 rank_strategy.cpp:63] RAW: create predictor PredictLinearModel success
I0909 10:45:00.390359 44 anyq_strategy.cpp:50] RAW: anyq init success
I0909 10:45:00.390359 44 session_data_factory.cpp:67] RAW: session data init success!!!
I0909 10:45:00.390359 44 utils.cpp:590] RAW: in json_to_analysis_item, query:需要使用什么账号登录
I0909 10:45:00.390359 44 utils.cpp:594] RAW: in json_to_analysis_item, type:(null)
I0909 10:45:00.390359 44 analysis_strategy.cpp:115] RAW: analysis_analysis size: 1
I0909 10:45:00.390359 44 analysis_strategy.cpp:130] RAW: before use analysis strategy's analysis_result
I0909 10:45:00.390359 44 utils.cpp:609] RAW: query:需要使用什么账号登录
I0909 10:45:00.390359 44 utils.cpp:611] RAW: tokens_basic size is 0
I0909 10:45:00.390359 44 analysis_strategy.cpp:148] RAW: method_process method_wordseg start
I0909 10:45:00.390359 44 analysis_strategy.cpp:154] RAW: method_process method_wordseg sucess
I0909 10:45:00.390359 44 analysis_strategy.cpp:163] RAW: after use analysis strategy's analysis_result
I0909 10:45:00.390359 44 utils.cpp:609] RAW: query:需要使用什么账号登录
I0909 10:45:00.390359 44 utils.cpp:611] RAW: tokens_basic size is 5
I0909 10:45:00.390359 44 utils.cpp:614] RAW: tokens_basic buffer:需要 length:6 offset:0 analysis_term_weight:0.200000
I0909 10:45:00.390359 44 utils.cpp:614] RAW: tokens_basic buffer:使用 length:6 offset:6 analysis_term_weight:0.200000
I0909 10:45:00.390359 44 utils.cpp:614] RAW: tokens_basic buffer:什么 length:6 offset:12 analysis_term_weight:0.200000
I0909 10:45:00.390359 44 utils.cpp:614] RAW: tokens_basic buffer:账号 length:6 offset:18 analysis_term_weight:0.200000
I0909 10:45:00.390359 44 utils.cpp:614] RAW: tokens_basic buffer:登录 length:6 offset:24 analysis_term_weight:0.200000
I0909 10:45:00.390359 44 equal_solr_q_builder.cpp:48] RAW: equal solr_fetch_q=question:需要使用什么账号登录
I0909 10:45:00.390359 44 term_retrieval.cpp:109] RAW: solr_fetch_q=question:需要使用什么账号登录
I0909 10:45:00.390359 44 term_retrieval.cpp:119] RAW: url = http://127.0.0.1:8900/solr/collection1/select
I0909 10:45:00.390359 44 http_client.cpp:84] RAW: para_url: fl=id,question,answer&q=question%3A%E9%9C%80%E8%A6%81%E4%BD%BF%E7%94%A8%E4%BB%80%E4%B9%88%E8%B4%A6%E5%8F%B7%E7%99%BB%E5%BD%95&rows=15&wt=json
W0909 10:45:00.390359 44 http_client.cpp:94] RAW: curl_easy_perform Failed[Couldn't connect to server];
W0909 10:45:00.390359 44 term_retrieval.cpp:205] RAW: solr_request failed.
E0909 10:45:00.390359 44 retrieval_strategy.cpp:122] RAW: plugin term_recall_1 retrieval error
E0909 10:45:00.390359 44 anyq_strategy.cpp:69] RAW: retrieval module process error
E0909 10:45:00.390359 44 http_service_impl.cpp:133] RAW: anyq run_strategy failed!

这是什么引起的？

Can we use SimNet to train our model?

can we fine tune the simnet model or train from scratch if we have our own datasets.

run_server 执行时出错，重复定义参数

运行报错：

➜  build git:(master) ✗ ./run_server
ERROR: flag 'v' was defined more than once (in files 'src/vlog_is_on.cc' and 'src/butil/logging.cc').

运行 paddle_simnet.py 出现 RuntimeError: boost::bad_get: failed value get using boost::get

Paddle 版本信息：docker 安装的 paddlepaddle-gpu==0.14.0

在 AnyQ/tools/simnet/train/paddle 目录下运行 sh run_train.sh，出现错误：

Traceback (most recent call last):
  File "paddle_simnet.py", line 178, in <module>
    train(conf_dict)
  File "paddle_simnet.py", line 91, in train
    main_program=fluid.default_main_program())
  File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/parallel_executor.py", line 155, in __init__
    build_strategy, num_trainers, trainer_id)
RuntimeError: boost::bad_get: failed value get using boost::get

Solution：
因为错误发生在 fluid.ParallelExecutor 函数，我预计是因为 docker 的 Paddle 是单 GPU 运行的。然后修改把 ParallelExecutor 相关的代码改成 Executor，如下：

    ## Get and run executor
    #parallel_executor = fluid.ParallelExecutor(
    #    use_cuda=False, loss_name=avg_cost.name,
    #    main_program=fluid.default_main_program())
    ## Get device number
    #device_count = parallel_executor.device_count
    #logging.info("device count: %d" % device_count)
    # run train
    logging.info("start train process ...")
    for epoch_id in range(conf_dict["epoch_num"]):
        losses = []
        # Get batch data iterator
        batch_data = paddle.batch(reader, conf_dict["batch_size"], drop_last=False)
        start_time = time.time()
        for iter, data in enumerate(batch_data()):
            #if len(data) < device_count:
            #    continue
            #avg_loss = parallel_executor.run(
            #    [avg_cost.name], feed=feeder.feed(data))
            avg_loss = executor.run(
                fetch_list=[avg_cost.name], feed=feeder.feed(data))
            print("epoch: %d, iter: %d, loss: %f" %
                (epoch_id, iter, np.mean(avg_loss[0])))
            losses.append(np.mean(avg_loss[0]))
        end_time = time.time()

然后运行没问题了。供参考。

重排序模块增加bm25，计算失败

为重排序增加bm25相关性计算遇到问题，请教大神们是否操作上有错

操作过程

在rank.conf中增加，并修改rank_weignts，启动脚本

matching_config {
    name : "bm25_sim"
    type : "BM25Similarity"
    output_num : 1
    rough : false
}

出错记录

1. 提交查询请求，日志中看到不到bm25的计算结果
2. 运行一段时间后报错，错误如下

Segmentation fault

how to run demo

According to the doc to compile and config the project, it seems nothing unhappy happen. From the following step, I have no idea:

How can I test the demo ? How can I get the right host and port ?

如何不使用./run_server 命令启动AnyQ?

目前打算在docker内使用crontab 定时重启 AnyQ ,以解决语义不能热加载的情况

先在crontab加上

*/1 * * * * nohup /bin/sh /opt/AnyQ/build/semantic.sh >> /tmp/anq.log &

semantic.sh 脚本如下

#!/bin/bash
source /etc/profile

sh solr_script/build_index.sh solr_script/sample_docs
echo "finish write solr"
python solr_script/make_json.py solr_script/sample_docs faq/schema_format faq/faq_json
awk -F "\t" '{print ++ind"\t"$0}' faq/faq_json > faq/faq_json.index
./annoy_index_build_tool example/conf/ example/conf/analysis.conf faq/faq_json.index 128 10 semantic.annoy 1>std 2>err
\cp -rf faq/faq_json.index semantic.annoy example/conf
echo "finish write semantic.annoy"

echo "restart anyq"

echo "stoping run_server"
unset GREP_OPTIONS
pid=$(ps axuf|grep "run_server"|grep -v grep |awk '{print $2}')
if [[ ! -z $pid ]]
then
        kill -3 $pid
        sleep 5
        kill -9 $pid
fi
echo "stop run_server"

./run_server

目前卡在 run_server 这步 ,请问该如何启动呢?

请问一下这里面有预训练的模型吗

谢谢