userver-framework / userver Goto Github PK
View Code? Open in Web Editor NEWProduction-ready C++ Asynchronous Framework with rich functionality
Home Page: https://userver.tech
License: Apache License 2.0
Production-ready C++ Asynchronous Framework with rich functionality
Home Page: https://userver.tech
License: Apache License 2.0
В ряде случаев WebSocket имеет преимущество перед HTTP. Было бы здорово увидеть его поддержку в userver.
I feel like NonFifo
is very unclear in terms of what a queue actually represents and leaves too wide of a room for an interpretation: stack is NonFifo
, dequeing random element is NonFifo
etc.
Maybe something in the lines of CasualOrder
/NonLinearizable
would fit better?
To add to that, NonFifoSp[s|m]cQueue
is actually fifo, because it serializes writes through a single producer token, as suggested in moodycamel documentation:
However, if for some reason you do extra explicit synchronization between the two producer threads yourself, thus defining a total order between enqueue operations, you might expect that the elements would come out in the same total order, which is a guarantee my queue does not offer. At that point, though, there semantically aren't really two separate producers, but rather one that happens to be spread across multiple threads. In this case, you can still establish a total ordering with my queue by creating a single producer token, and using that from both threads to enqueue (taking care to synchronize access to the token, of course, but there was already extra synchronization involved anyway).
There also exists https://github.com/cameron314/readerwriterqueue, which might be a better fit for spsc queue version
Thanks for the long awaited framework. You have done a great job!
I would really like to get support for the async amqp protocol,
are there any plans to add it in the near future? Thank you!
При билде тестового сервиса созданного из шаблона получаю такую ошибку:
File "/media/sf_userver/test_service/third_party/userver/plugins/external_deps/impl/cmake_generator.py", line 211, in
main()
File "/media/sf_userver/test_service/third_party/userver/plugins/external_deps/impl/cmake_generator.py", line 206, in main
with open(file_path, 'w') as output_file:
FileNotFoundError: [Errno 2] No such file or directory: '/media/sf_userver/test_service/build_debug/cmake_generated/FindIconv::Iconv.cmake'
CMake Error at third_party/userver/CMakeLists.txt:70 (message):
Generating cmake files failed with exit code: 1
Я вижу, что не генерится какой-то файл с путями к какой-то либе, но её название абсолютное невзрачное. Можете подсказать, что я делаю не так?
Looking at flamegraphs from #141 (comment) one can see that unordered_map
takes ~10-11% off total time, with most of it coming from http internals (and some from TimeStorage
).
There is an idea that it could very well be replaced with flat map, that would fall back to unordered_map after some threshold, and chances are it would perform better - not only std::unordered_map is pretty slow and linear search might perform better for small number of keys, but there also is that HashDOS-preventing hash, which ain't the fastest. Also my experience tells me that it's not that common to have, say, 50+ headers in http.
What is also promising is that this abstract HttpHeadersMap
could be implemented via on stack buffer containing data in http-format right away! - just add some array of string_views which indexes into that buffer. Not sure whether that would work for requests, but should be possible for responses, and for responses we could just pass it's content in Socket::SendAll
as is
In this code coroutine won't be canceled if TCP connection is a live during shutdown process. Coroutine is waiting for wake-up, ev thread is waiting for socket events. The only way to close coroutine is to send something through connection or close it on remote site.
Expected behavior is to wake up all tasks and throw canceled exception if manager is shuting down.
static engine::Task task;
class StuckedTCPServer final : public userver::components::LoggableComponentBase {
public:
static constexpr std::string_view kName = "stucked-tcp-server";
StuckedTCPServer(const userver::components::ComponentConfig& component_config,
const userver::components::ComponentContext& component_context)
: userver::components::LoggableComponentBase(component_config, component_context) {
userver::server::net::ListenerConfig listener;
listener.port = 8080;
task = utils::Async(
"listener",
[](auto socket) {
while (!engine::current_task::ShouldCancel()) {
auto s = socket.Accept({});
std::array<char, 32> buf;
s.RecvAll(buf.data(), buf.size(), {});
}
},
userver::server::net::CreateSocket(listener));
}
void OnAllComponentsAreStopping() override {
// close active connections
}
};
int main(int argc, char* argv[]) {
auto component_list = components::ComponentList()
.Append<os_signals::ProcessorComponent>()
.Append<components::Logging>()
.Append<components::Tracer>()
.Append<components::ManagerControllerComponent>()
.Append<components::StatisticsStorage>()
.Append<components::DynamicConfig>()
.Append<components::DynamicConfigFallbacks>()
.Append<StuckedTCPServer>();
return userver::utils::DaemonMain(argc, argv, component_list);
}
Please improve the documentation about the pre-requirements packages for macOS
here
Now, the configure process is not easy for macOS
. For instance, I faced some problems when realpath
wasn't installed on my PC. So, I had to install coreutils
.
Setup to reproduce:
sudo /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
nc -l 5672
kill -9
your haproxy process from step 2read_ret = 0
and ssl_error = 2
taking ~100% of CPUitrofimow@itrofimow-RedmiBook-16:~$ openssl version
OpenSSL 1.1.1 11 Sep 2018
itrofimow@itrofimow-RedmiBook-16:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"
Полагаю, такси все еще не отпускает)
https://github.com/userver-framework/userver/blob/develop/core/include/userver/dynamic_config/client/component.hpp 56
In .gitmodules
there are links to the internal Yandex repository inaccessible from the outer web
Hello. I would like to report an issue on Arch Linux.
Build system was generated with this command, as suggested in docs:
cmake -DUSERVER_FEATURE_PATCH_LIBPQ=0 -DCMAKE_BUILD_TYPE=Release ..
Build error:
[ 65%] Linking CXX executable userver-samples-postgres_service
/usr/sbin/ld: cannot find -lPostgreSQLInternal_LIBRARIES_libpq_a-NOTFOUND: No such file or directory
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [samples/postgres_service/CMakeFiles/userver-samples-postgres_service.dir/build.make:128: samples/postgres_service/userver-samples-postgres_service] Error 1
make[1]: *** [CMakeFiles/Makefile2:3265: samples/postgres_service/CMakeFiles/userver-samples-postgres_service.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
libpq
was not found at configure process, but build system generation step was not failed and the issue was not reported. Tested on two different machines. I tries to install / reinstall packages from docs both with pacman and aur tool.My system info:
clang version 14.0.6
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
IMAGE_VERSION=2022.02.01
cmake -DCMAKE_BUILD_TYPE=Release ..
(without turning off patch for libpq
).This time generating build system fails with the following output:
-- Installed version postgresql-libs: 14.5
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could not find `PostgreSQL` package.
Debian: sudo apt update && sudo apt install libpq-dev
MacOS: brew install postgresql@14
Fedora: sudo dnf install postgresql-static
ArchLinux: sudo pacman -S postgresql-libs
(missing: PostgreSQL_LIBRARIES) (Required is at least version "12")
Call Stack (most recent call first):
/usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
build_release/cmake_generated/FindPostgreSQL.cmake:69 (find_package_handle_standard_args)
postgresql/pq-extra/CMakeLists.txt:14 (find_package)
The required package is installed.
❯ sudo pacman -S postgresql-libs
warning: postgresql-libs-14.5-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...
Packages (1) postgresql-libs-14.5-1
Total Installed Size: 6.90 MiB
Net Upgrade Size: 0.00 MiB
:: Proceed with installation? [Y/n] ^
This may relate to #88
Ubuntu 20.04
while installing following error after given make -j$(nproc)
[ 52%] Built target netcat
[ 52%] Building CXX object tools/congestion_control_emulator/CMakeFiles/congestion_control_emulator.dir/congestion_control_emulator.cpp.o
[ 52%] Linking CXX executable congestion_control_emulator
[ 52%] Linking CXX executable dns-resolver
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans22.ltrans.o: in function `bool boost::regex_search<char const*, char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >(char const*, char const*, boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > > const&, boost::regex_constants::_match_flags) [clone .constprop.0]':
/usr/include/boost/regex/v4/perl_matcher.hpp:386: undefined reference to `boost::re_detail_106700::perl_matcher<char const*, std::allocator<boost::sub_match<char const*> >, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::construct_init(boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > > const&, boost::regex_constants::_match_flags)'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans22.ltrans.o: in function `bool boost::regex_search<char const*, char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >(char const*, char const*, boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > > const&, boost::regex_constants::_match_flags) [clone .constprop.0]':
/usr/include/boost/regex/v4/perl_matcher_non_recursive.hpp:107: undefined reference to `boost::re_detail_106700::get_mem_block()'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans22.ltrans.o: in function `bool boost::regex_search<char const*, char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >(char const*, char const*, boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > > const&, boost::regex_constants::_match_flags) [clone .constprop.0]':
/usr/include/boost/regex/v4/perl_matcher_common.hpp:305: undefined reference to `boost::re_detail_106700::verify_options(unsigned int, boost::regex_constants::_match_flags)'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans22.ltrans.o: in function `bool boost::regex_search<char const*, char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >(char const*, char const*, boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > > const&, boost::regex_constants::_match_flags) [clone .constprop.0]':
/usr/include/boost/regex/v4/perl_matcher_non_recursive.hpp:115: undefined reference to `boost::re_detail_106700::put_mem_block(void*)'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans22.ltrans.o: in function `bool boost::regex_search<char const*, char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >(char const*, char const*, boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > > const&, boost::regex_constants::_match_flags) [clone .constprop.0] [clone .cold]':
/usr/include/boost/regex/v4/perl_matcher_non_recursive.hpp:115: undefined reference to `boost::re_detail_106700::put_mem_block(void*)'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans24.ltrans.o: in function `char const* boost::re_detail_106700::re_is_set_member<char const*, char, boost::regex_traits<char, boost::cpp_regex_traits<char> >, unsigned int>(char const*, char const*, boost::re_detail_106700::re_set_long<unsigned int> const*, boost::re_detail_106700::regex_data<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > > const&, bool)':
/usr/include/boost/regex/v4/cpp_regex_traits.hpp:966: undefined reference to `boost::re_detail_106700::cpp_regex_traits_implementation<char>::transform_primary[abi:cxx11](char const*, char const*) const'
/usr/bin/ld: /usr/include/boost/regex/v4/cpp_regex_traits.hpp:962: undefined reference to `boost::re_detail_106700::cpp_regex_traits_implementation<char>::transform[abi:cxx11](char const*, char const*) const'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans24.ltrans.o: in function `void boost::re_detail_106700::raise_error<boost::regex_traits_wrapper<boost::regex_traits<char, boost::cpp_regex_traits<char> > > >(boost::regex_traits_wrapper<boost::regex_traits<char, boost::cpp_regex_traits<char> > > const&, boost::regex_constants::error_type)':
/usr/include/boost/regex/v4/cpp_regex_traits.hpp:447: undefined reference to `boost::re_detail_106700::get_default_error_string(boost::regex_constants::error_type)'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans24.ltrans.o: in function `void boost::re_detail_106700::raise_error<boost::regex_traits_wrapper<boost::regex_traits<char, boost::cpp_regex_traits<char> > > >(boost::regex_traits_wrapper<boost::regex_traits<char, boost::cpp_regex_traits<char> > > const&, boost::regex_constants::error_type)':
/usr/include/boost/regex/pattern_except.hpp:75: undefined reference to `boost::re_detail_106700::raise_runtime_error(std::runtime_error const&)'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans24.ltrans.o: in function `boost::re_detail_106700::perl_matcher<char const*, std::allocator<boost::sub_match<char const*> >, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::extend_stack()':
/usr/include/boost/regex/v4/perl_matcher_non_recursive.hpp:233: undefined reference to `boost::re_detail_106700::get_mem_block()'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans24.ltrans.o: in function `boost::re_detail_106700::perl_matcher<char const*, std::allocator<boost::sub_match<char const*> >, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::match_then()':
/usr/include/boost/regex/v4/perl_matcher_non_recursive.hpp:233: undefined reference to `boost::re_detail_106700::get_mem_block()'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans24.ltrans.o: in function `boost::re_detail_106700::perl_matcher<char const*, std::allocator<boost::sub_match<char const*> >, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::match_commit()':
/usr/include/boost/regex/v4/perl_matcher_non_recursive.hpp:233: undefined reference to `boost::re_detail_106700::get_mem_block()'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans24.ltrans.o: in function `boost::re_detail_106700::perl_matcher<char const*, std::allocator<boost::sub_match<char const*> >, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::match_toggle_case()':
/usr/include/boost/regex/v4/perl_matcher_non_recursive.hpp:233: undefined reference to `boost::re_detail_106700::get_mem_block()'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans25.ltrans.o: in function `boost::re_detail_106700::perl_matcher<char const*, std::allocator<boost::sub_match<char const*> >, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::unwind_extra_block(bool)':
/usr/include/boost/regex/v4/perl_matcher_non_recursive.hpp:1368: undefined reference to `boost::re_detail_106700::put_mem_block(void*)'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans25.ltrans.o: in function `userver::utils::regex_match(std::basic_string_view<char, std::char_traits<char> >, userver::utils::regex const&)':
/usr/include/boost/regex/v4/perl_matcher.hpp:386: undefined reference to `boost::re_detail_106700::perl_matcher<char const*, std::allocator<boost::sub_match<char const*> >, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::construct_init(boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > > const&, boost::regex_constants::_match_flags)'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans25.ltrans.o: in function `userver::utils::regex_match(std::basic_string_view<char, std::char_traits<char> >, userver::utils::regex const&)':
/usr/include/boost/regex/v4/perl_matcher_non_recursive.hpp:107: undefined reference to `boost::re_detail_106700::get_mem_block()'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans25.ltrans.o: in function `userver::utils::regex_match(std::basic_string_view<char, std::char_traits<char> >, userver::utils::regex const&)':
/usr/include/boost/regex/v4/perl_matcher_common.hpp:219: undefined reference to `boost::re_detail_106700::verify_options(unsigned int, boost::regex_constants::_match_flags)'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans25.ltrans.o: in function `userver::utils::regex_match(std::basic_string_view<char, std::char_traits<char> >, userver::utils::regex const&)':
/usr/include/boost/regex/v4/perl_matcher_non_recursive.hpp:115: undefined reference to `boost::re_detail_106700::put_mem_block(void*)'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans25.ltrans.o: in function `userver::utils::regex_match(std::basic_string_view<char, std::char_traits<char> >, userver::utils::regex const&) [clone .cold]':
/usr/include/boost/regex/v4/perl_matcher_non_recursive.hpp:115: undefined reference to `boost::re_detail_106700::put_mem_block(void*)'
/usr/bin/ld: /tmp/userver-universal_unittest.fzUonP.ltrans26.ltrans.o: in function `userver::fs::blocking::(anonymous namespace)::RemoveDirectory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
/usr/include/boost/system/error_code.hpp:461: undefined reference to `boost::system::system_category()'
collect2: error: ld returned 1 exit status
make[2]: *** [userver/universal/CMakeFiles/userver-universal_unittest.dir/build.make:1441: userver/universal/userver-universal_unittest] Error 1
make[1]: *** [CMakeFiles/Makefile2:3212: userver/universal/CMakeFiles/userver-universal_unittest.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 52%] Built target httpclient_perf
[ 52%] Built target congestion_control_emulator
[ 52%] Built target dns-resolver
make: *** [Makefile:146: all] Error 2
Running socket benchmarks (--benchmark_filter='socket*'
) produces this tsan output
socket_tsan.txt
Is it FP or is some bug actually lurking?
In file included from /usr/include/c++/12.1.0/x86_64-pc-linux-gnu/bits/os_defines.h:39,
from /usr/include/c++/12.1.0/x86_64-pc-linux-gnu/bits/c++config.h:655,
from /usr/include/c++/12.1.0/type_traits:38,
from /usr/include/c++/12.1.0/optional:37,
from /root/code/userver/clickhouse/include/userver/storages/clickhouse/io/columns/nullable_column.hpp:7,
from /root/code/userver/clickhouse/src/storages/clickhouse/io/columns/nullable_column.cpp:1:
/usr/include/features.h:412:4: warning: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Wcpp]
412 | # warning _FORTIFY_SOURCE requires compiling with optimization (-O)
| ^~~~~~~
/root/code/userver/clickhouse/include/userver/storages/clickhouse/io/columns/nullable_column.hpp: In member function ‘userver::storages::clickhouse::io::columns::NullableColumn<T>::NullableDataHolder userver::storages::clickhouse::io::columns::NullableColumn<T>::NullableDataHolder::operator++(int)’:
/root/code/userver/clickhouse/include/userver/storages/clickhouse/io/columns/nullable_column.hpp:94:25: error: ‘exchange’ is not a member of ‘std’
94 | old.has_value_ = std::exchange(has_value_, false);
| ^~~~~~~~
make[3]: *** [userver/clickhouse/CMakeFiles/userver-clickhouse.dir/build.make:328: userver/clickhouse/CMakeFiles/userver-clickhouse.dir/src/storages/clickhouse/io/columns/nullable_column.cpp.o] Error 1
make[2]: *** [CMakeFiles/Makefile2:1663: userver/clickhouse/CMakeFiles/userver-clickhouse.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:2325: samples/clickhouse_service/CMakeFiles/userver-samples-clickhouse_service.dir/rule] Error 2
make: *** [Makefile:868: userver-samples-clickhouse_service] Error 2
Seems like nullable_column.hpp
does not include utility
header
userver/core/include/userver/server/http/form_data_arg.hpp
Lines 15 to 22 in cb8efc3
Непонятное назначение двух инверсий. Ведь value.data()
, указатель и сам скастуется в bool
при необходимости. Это преобразование не считается чем-либо постыдным и не вызывает предупреждений ни у каких компиляторов.
Не уверен, что проброс value.data()
это лучший способ реализовать operator bool()
для данного типа, ведь поле value
теоретически может быть пустой строкой:
form_data_arg.value = ""
И тогда operator bool()
вернет true
, хотя, возможно, должен вернуть false
. Я недостаточно хорошо знаю код, чтобы понять, какая семантика требуется.
Would be great to see the following features implemented in RabbitMQ driver:
Continuation of #44, basically
First of all, thank you guys for this impressive piece of software.
As stated in the title, i'm a bit concerned about the lack of comparison benchmarks: guess its safe to assume that if someone wants a c++ web-framework they mostly strive for c++ performance, and only after that for simplicity of Python, coroutine model of Go or other things, however there's no evidence of "speed of c++" aside from using c++, and that kinda questions the point of the framework (imo)
hi:
When I try to build userver, got errors below, It seems something error with libabsl?
operating system: Arch linux
cc: g++ (GCC) 12.1.1 20220730
cmake command: cmake -DUSERVER_FEATURE_PATCH_LIBPQ=0 -DCMAKE_BUILD_TYPE=Release ..
make output(errors):
/usr/bin/ld: libuserver-samples-grpc_service_proto.a(greeter.grpc.pb.cc.o): undefined reference to symbol '_ZN4absl12lts_202206235MutexD1Ev'
/usr/bin/ld: /usr/lib/libabsl_synchronization.so.2206.0.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [samples/grpc_service/CMakeFiles/userver-samples-grpc_service.dir/build.make:129: samples/grpc_service/userver-samples-grpc_service] Error 1
Hello,
We have "build nodes" that are "docker nodes" with configurations tmpfs - 2G (for /tmp), ramfs - 8G (for build folder). For this project, our configuration is:
docker run -it \
--rm \
--name userver_build_release \
--mount type=tmpfs,destination=/tmp,tmpfs-size=2147483648 \
--volume /mnt/ramfs/userver:/userver/build_release \
ubuntu:22.04
And tried build:
apt update && apt install -y git
cd userver/
git init
git remote add origin https://github.com/userver-framework/userver.git
git pull
git checkout develop -f
# Environments for tzdata
export DEBIAN_FRONTEND=noninteractive
export TZ=Etc/UTC
# https://userver.tech/d1/d03/md_en_userver_tutorial_build.html#installationinstructions
apt install -y $(cat scripts/docs/en/deps/ubuntu-22.04.md | tr '\n' ' ')
git config --global --add safe.directory $(pwd)/third_party/clickhouse-cpp
mkdir build_release
cd build_release
cmake -DCMAKE_BUILD_TYPE=Release ..
# Check available size in /tmp
df -h /tmp
# Build project
make -j12
We got errors :
lto-wrapper: fatal error: write: No space left on device
compilation terminated.
lto1: fatal error: error writing to /tmp/ccwzjvhU.s: No space left on device
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
lto-wrapper: fatal error: /usr/bin/c++ returned 1 exit status
compilation terminated.
collect2: error: ld returned 1 exit status
make[2]: *** [samples/clickhouse_service/CMakeFiles/userver-samples-clickhouse_service.dir/build.make:129: samples/clickhouse_service/userver-samples-clickhouse_service] Error 1
make[1]: *** [CMakeFiles/Makefile2:3842: samples/clickhouse_service/CMakeFiles/userver-samples-clickhouse_service.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 66%] Building CXX object userver/core/CMakeFiles/userver-utest.dir/testing/src/utest/dns_server_mock.cpp.o
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[2]: *** [samples/http_caching/CMakeFiles/userver-samples-http_caching.dir/build.make:122: samples/http_caching/userver-samples-http_caching] Error 1
make[1]: *** [CMakeFiles/Makefile2:3397: samples/http_caching/CMakeFiles/userver-samples-http_caching.dir/all] Error 2
Of course we can disable tests, examples and USERVER_FEATURE_* but it looks strange
Hello and congratulations on open sourcing this amazing work!
There's a section on the page about dynamic configs that mentions userver-dynconf:
Note that there is a ready to use opensource uservice-dynconf dynamic configs service. Use it for your projects or just disable dynamic config updates and keep developing without a supplementary service.
Following the link to uservice-dynconf yields a GitHub 404 page, which is probably not expected.
Some issues with conan and utest
Here is how to reproduce this issue
USERVER_FEATURE_UTEST=ON
userver::userver
or even CONAN_LIB::userver_userver-utest
userver/utest/utest.hpp
doesn't exist.using_namespace_userver.hpp
file in userver/utest
CMake Functions such as add_google_tests
and userver_testsuite_add
also cannot be seen.
Часто приходится реализовывать запросы к БД, которые в зависимости от переданных параметров должны добавлять в запрос к БД дополнительные условия - фильтры.
Например, если передали строковый параметр, то нужно поискать в строковых полях, если передали коллекцию идентификаторов, то поискать с вхождением какого-то поля в эту коллекцию( массив).
Для этих целей удобно использовать QueryBuilder по типу такого:
https://docs.rs/sqlx/latest/sqlx/struct.QueryBuilder.html
Фактически он просто склеивает запрос, позволяя вставлять аргументы с эскейпингом для защиты от иньекций.
Добавляет escaped значение фильтра:
builder.bind(filter)
Вставляет строку в запрос как есть:
buidler.bind(" AND value IN ")
ну и другие удобные методы.
Есть ли что-то подобное в userver? Я не нашел. Если нет, было бы классно реализовать
Do you have a tcp echo server that I can run with? So I can add the figures into this list.
Hi!
Thanks for the library! I guess now it is kinda proven that userver really exists :)
Regarding the problem. In README you have a link to the documentation: See the docs for more info. Please change the link to the actual documentation which is public accessible.
Thanks in advance!
make[3]: *** No rule to make target 'samples/grpc_service/proto/contracts/common_client.usrv.pb.cpp', needed by 'samples/grpc_service/CMakeFiles/userver-samples-grpc_service_proto.dir/proto/contracts/common_client.usrv.pb.cpp.o'. Stop.
If you add any rpc declaration to the .proto file, it will be built
I tried to compile userver on CentOS 7.8 based on 4.19 kernel and got the following error, on the same machine using https://github.com/userver-framework/docker-userver-build-base/blob/develop/ The compilation in the dockerfile generated by Dockerfile does not have this problem.
$ make
[ 1%] Built target userver-uboost-coro
[ 1%] Built target fmt
[ 6%] Built target c-ares
[ 18%] Built target libcurl
[ 32%] Built target cryptopp-static
[ 33%] Built target cctz
[ 33%] Built target userver-stacktrace
Scanning dependencies of target userver-core
[ 33%] Building CXX object userver/core/CMakeFiles/userver-core.dir/__/shared/src/crypto/base.cpp.o
[ 33%] Building CXX object userver/core/CMakeFiles/userver-core.dir/__/shared/src/crypto/base64.cpp.o
[ 33%] Building CXX object userver/core/CMakeFiles/userver-core.dir/__/shared/src/crypto/certificate.cpp.o
[ 33%] Building CXX object userver/core/CMakeFiles/userver-core.dir/__/shared/src/crypto/hash.cpp.o
[ 33%] Building CXX object userver/core/CMakeFiles/userver-core.dir/__/shared/src/crypto/helpers.cpp.o
[ 34%] Building CXX object userver/core/CMakeFiles/userver-core.dir/__/shared/src/crypto/private_key.cpp.o
[ 34%] Building CXX object userver/core/CMakeFiles/userver-core.dir/__/shared/src/crypto/public_key.cpp.o
[ 34%] Building CXX object userver/core/CMakeFiles/userver-core.dir/__/shared/src/crypto/signers.cpp.o
[ 34%] Building CXX object userver/core/CMakeFiles/userver-core.dir/__/shared/src/crypto/verifiers.cpp.o
[ 34%] Building CXX object userver/core/CMakeFiles/userver-core.dir/__/shared/src/formats/json/impl/mutable_value_wrapper.cpp.o
In file included from userver/core/../shared/include/userver/formats/json/impl/mutable_value_wrapper.hpp:9,
from userver/shared/src/formats/json/impl/mutable_value_wrapper.cpp:1:
userver/core/../shared/include/userver/utils/fast_pimpl.hpp: In instantiation of 'static void userver::utils::FastPimpl<T, Size, Alignment, Strict>::Validate() [with long unsigned int ActualSize = 64; long unsigned int ActualAlignment = 8; T = userver::formats::json::impl::MutableValueWrapper::Impl; long unsigned int Size = 88; long unsigned int Alignment = 8; bool Strict = true]':
userver/core/../shared/include/userver/utils/fast_pimpl.hpp:86:36: required from 'userver::utils::FastPimpl<T, Size, Alignment, Strict>::~FastPimpl() [with T = userver::formats::json::impl::MutableValueWrapper::Impl; long unsigned int Size = 88; long unsigned int Alignment = 8; bool Strict = true]'
userver/shared/src/formats/json/impl/mutable_value_wrapper.cpp:48:1: required from here
userver/core/../shared/include/userver/utils/fast_pimpl.hpp:95:27: error: static assertion failed: invalid Size: Size == sizeof(T) failed
95 | static_assert(!Strict || Size == ActualSize,
| ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
userver/core/../shared/include/userver/utils/fast_pimpl.hpp:95:27: note: '(false || (88 == 64))' evaluates to false
make[2]: *** [userver/core/CMakeFiles/userver-core.dir/build.make:317: userver/core/CMakeFiles/userver-core.dir/__/shared/src/formats/json/impl/mutable_value_wrapper.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:858: userver/core/CMakeFiles/userver-core.dir/all] Error 2
make: *** [Makefile:161: all] Error 2
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz
Stepping: 7
CPU MHz: 2099.996
CPU max MHz: 2100.0000
CPU min MHz: 800.0000
BogoMIPS: 4200.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 28160K
NUMA node0 CPU(s): 0-19,40-59
NUMA node1 CPU(s): 20-39,60-79
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
$ cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)
$ uname -a
Linux no.docker 4.19.136 #1 SMP Fri Nov 6 17:54:05 CST 2020 x86_64 x86_64 x86_64 GNU/Linux
$ c++ -v
Using built-in specs.
COLLECT_GCC=c++
COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-11/root/usr/libexec/gcc/x86_64-redhat-linux/11/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/opt/rh/devtoolset-11/root/usr --mandir=/opt/rh/devtoolset-11/root/usr/share/man --infodir=/opt/rh/devtoolset-11/root/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --with-default-libstdcxx-abi=gcc4-compatible --enable-plugin --enable-initfini-array --with-isl=/builddir/build/BUILD/gcc-11.2.1-20220127/obj-x86_64-redhat-linux/isl-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 11.2.1 20220127 (Red Hat 11.2.1-9) (GCC)
$ cmake --version
cmake3 version 3.17.5
CMake suite maintained and supported by Kitware (kitware.com/cmake).
$ cmake
-DCMAKE_INSTALL_PREFIX=$HOME/local \
-DBUILD_EXAMPLES=0 \
-DOPENSSL_ROOT_DIR=/usr/include/openssl11/ \
-DBoost_INCLUDE_DIR=/usr/include/boost169/ \
-DBoost_LIBRARY_DIR=/usr/lib64/boost169/ \
\
-DUSERVER_DOWNLOAD_PACKAGES=1 \
-DUSERVER_FEATURE_GRPC=0 \
-DUSERVER_FEATURE_REDIS=0 \
-DUSERVER_FEATURE_MONGODB=0 \
-DUSERVER_FEATURE_POSTGRESQL=0 \
-DUSERVER_FEATURE_CLICKHOUSE=0 \
-DUSERVER_FEATURE_STACKTRACE=0 \
-DUSERVER_FEATURE_SPDLOG_TCP_SINK=0 \
..
Unhandled exception in components::Run: Cannot load fallback taxi config: Error opening '/etc/hello_service/dynamic_config_fallback.json'
should be "Cannot load fallback dynamic config"
There exists https://github.com/cameron314/readerwriterqueue, which might be a better fit for concurrent::SpscQueue
– it has to be faster than full-featured moodycamel::ConcurrentQueue
and it should also require less memory.
There is some confusion between readme and docs:
In readme it's stated that
It only supports a two-thread use case (one consuming, and one producing). The threads can't switch roles, ...
However inline docs elaborate that, leaving a room for experimentation.
Userver have a lot warnings with suggested disabling USERVER_CHECK_PACKAGE_VERSIONS
Also have a lot of problems on linkage time like #19 but different libs
I think its a bad way to mute this warnings.
We can improve situation by addition pacman version detector to DetectVersion.cmake
There are some situations when a resource is accessed concurrently, but concurrency factor is limited by 2, - implications of socket r/w duality come to mind, - and engine::Mutex
seems like a go to choice (and it is), however it operates over full-featured WaitList
, when in this specific case WaitListLight
would do, since there's at most 1 waiter at any given moment.
Doing -3 lines of code makes wonders when running TechEmpower pipelined plaintext benchmarks on 24-cores VM (16 worker threads, 4 ev threads, 6 threads for wrk):
with TCP_NODELAY:
Concurrency: 1024 for plaintext
wrk -H 'Host: tfb-server' -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 1024 --timeout 8 -t 24 http://tfb-server:8090/plaintext -s pipeline.lua -- 16
---------------------------------------------------------
Running 15s test @ http://tfb-server:8090/plaintext
24 threads and 1024 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 23.35ms 18.20ms 164.49ms 77.32%
Req/Sec 20.61k 3.52k 78.08k 81.24%
Latency Distribution
50% 19.16ms
75% 31.32ms
90% 45.71ms
99% 90.12ms
7417080 requests in 15.09s, 2.19GB read
Requests/sec: 491395.99
Transfer/sec: 148.56MB
without TCP_NODELAY:
Concurrency: 1024 for plaintext
wrk -H 'Host: tfb-server' -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 1024 --timeout 8 -t 24 http://tfb-server:8090/plaintext -s pipeline.lua -- 16
---------------------------------------------------------
Running 15s test @ http://tfb-server:8090/plaintext
24 threads and 1024 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 20.91ms 16.76ms 192.81ms 74.69%
Req/Sec 24.92k 3.27k 93.68k 88.27%
Latency Distribution
50% 15.95ms
75% 28.38ms
90% 46.78ms
99% 71.08ms
8957662 requests in 15.09s, 2.64GB read
Requests/sec: 593597.99
Transfer/sec: 179.45MB
I do understand that the case is somewhat specific, but hey, it's 20% speedup!
P.S. would be great if someone could recheck me here
Is utils::Async(...)
supposed to be callable from non-worker threads (say, ev
)?
I got the following code
UTEST(UtilsAsync, FromNonWorkerThread) {
auto& ev_thread = engine::current_task::GetEventThread();
auto& task_processor = engine::current_task::GetTaskProcessor();
engine::TaskWithResult<void> task;
ev_thread.RunInEvLoopSync([&task_processor, &task] {
task = utils::Async(task_processor, "just_a_task", []{});
});
task.Wait();
}
which asserts somewhere in Span
internals https://pastebin.com/raw/JdLrwQ5x,
and i feel like this behavior should either be documented or changed in a way that Span
gets instantiated in a thread that actually runs a task, not a thread that created it.
Or am i missing something?
P.S. engine::AsyncNoSpan
works as expected
I'm yet too see a correct usage of Detach
in user code, and there were (and could still be) some misuses even in the framework itself
People coming from .Net might think 'hey, it's Task.Run()!', people coming from Rust might find it similar to tokio::spawn
or the likes of, however there is one very important part missing: lifetimes.
.Net takes care of this with GC, rust does it's best with borrow checker, but neither is present in userver
Detaching a task basically gives it 'static
lifetime, thus capturing anything by reference without proper synchronization is straight UB, and even capturing by value is very dangerous - is there something holding a reference/pointer down the ownership chain? - You never know.
Talking about proper synchronization - there is BackgroundTaskStorage
, which got quite some love in performance regards recently, and it solves all the problems. Honestly i can't come up with a case that can not be solved with BTS
but could be solved by Detach
, in user code especially.
So my proposal is to either remove .Detach
from public API completely, or warn a user about all the problematic implications of detaching a task in documentation
Deadline is only checked if a socket would block and some waiting should be performed, however it's completely possible for read/write to go through without blocking.
One could expect that any attempt to do some work after deadline is reached would fail, but as of now it's not the case with Socket
You see, moodycamel::BlockingConcurrentQueue
spins here before going to kernel semaphore
and although it's preferable for maximum throughput in benchmarks,
it wastes CPU cycles in some real-life scenarios.
Since this commit there is a way to control this spinning behavior, which was very noticeable in some perfs, remember @apolukhin?
Hi,
On Readme documentation i see a block of code about postgres, but i don't see nothing related to the server.
Can you add to readme the source code to start the server or something like this?
Thanks.
Even though it's pretty convenient to build because of the great documentation you guys provide. But it's still helpful if it's available in the famous package management. I do think Conan and Vcpkg should be considered.
And by the way, thank you and it's a brilliant framework.
Due to moodycamel::ConcurrentQueue
implementation details engine::Yield
doesn't actually yield running thread back to the thread-pool:
since moodycamel consists of a set of thread-local FIFO queues and
every consumer checks for its paired producer queue first, when a coroutine-thread
calls engine::Yield
task continuation is scheduled "for himself",
thus when it tries to dequeue a task it will get the exact same continuation it just yielded
(it's not always true and moodycamel does rotate consumer<->producer pairing from time to time).
Here is an example of what i'm talking about, and sadly this situation has some real-life analogues:
Consider PostgreCache
: it uses engine::Yield
when copying a container or iterating fetched values, and that could take plenty of iterations, basically "blocking" current thread, even though it yields.
Imagine you have 2 cores, 2 worker-threads and 150RPS, where each request takes 10ms of CPU times:
2 threads should deal with it no problem, leaving 250ms of CPU time to spare,
however as soon as one thread gets into "yield-loop" second thread gets overwhelmed instantly
What's even worse in aforementioned example,
PostgreCache
does itself blocks a worker-thread when it deallocates cached data, but that's another problem i guess
First of all, thank you for userver. That's one small step for man, one giant leap for mankind.
I see you use postgres and mongo. Is there any chances for other dbms such as mysql, mssql or oracle?
Does it really require jinja2 at the CMake configuration step? I'd like to try building this without having to install any Python dependencies if possible.
Hey guys,
when connecting a dynamic-config-client-updater to a config server that's configured to use a postgresql db this error appears:
tskv timestamp=2022-08-26T17:33:41.127367 level=ERROR module=DoRun ( third_party/userver/core/src/components/run.cpp:164 ) task_id=0 thread_id=0x00007F1E59D55AC0 text=Loading failed: Error from 'dynamic-config-client-updater' updater: Field 'USERVER_HTTP_PROXY' is of a wrong type. Expected: stringValue, actual: objectValue (std::runtime_error) tskv timestamp=2022-08-26T17:33:41.129100 level=ERROR module=DaemonMain ( third_party/userver/core/src/utils/daemon_run.cpp:72 ) task_id=0 thread_id=0x00007F1E59D55AC0 text=Unhandled exception in components::Run: Error from 'dynamic-config-client-updater' updater: Field 'USERVER_HTTP_PROXY' is of a wrong type. Expected: stringValue, actual: objectValue Unhandled exception in components::Run: Error from 'dynamic-config-client-updater' updater: Field 'USERVER_HTTP_PROXY' is of a wrong type. Expected: stringValue, actual: objectValue
It's expecting a string but config_value is a json column.
The error doesn't happen if switching to the dynamic_config_fallback.json file there the value of USERVER_HTTP_PROXY is correctly a string.
Thanks
Это ошибка вызвана, предположительно, копипастом. Определение оператора неравенства
userver/core/include/userver/server/auth/user_scopes.hpp
Lines 44 to 46 in cb8efc3
Такое же, как и определение оператора равенства
userver/core/include/userver/server/auth/user_scopes.hpp
Lines 36 to 38 in cb8efc3
Очевидно, там подразумевалось !=
вместо ==
Is there a plan to add mariadb support in future..
На данный момент есть возможность передавать параметр типа std::vector в Execute и он эскейпится в значение массива.
Хорошо бы добавить возможность такого преобразования для google::protobuf::RepeatedField и для других коллекций.
В идеале было бы очень удобно, если бы была реализована функция Escape для любых типов у которых есть begin и end, которая работала бы как для std::vector.
I'm trying to build userver without test suite enabled (DUSERVER_FEATURE_TESTSUITE=OFF), but clickhouse_service
sample fails to build because of non-existent target testsuite-userver-samples-clickhouse_service
.
Related Cmake log:
-- Testsuite target testsuite-userver-samples-clickhouse_service is disabled
CMake Error at samples/clickhouse_service/CMakeLists.txt:7 (set_tests_properties):
set_tests_properties Can not find test to add properties to:
testsuite-userver-samples-clickhouse_service
-- Configuring incomplete, errors occurred!
See also "/home/popov/projects/github/userver/cmake-build-debug/CMakeFiles/CMakeOutput.log".
See also "/home/popov/projects/github/userver/cmake-build-debug/CMakeFiles/CMakeError.log".
[Failed to reload]
It would be nice if the sample would work without test suite enabled. Maybe some check can be added before using set_tests_properties
in clickhouse_service
Why this global variables are not constexpr ? Seems like std::string_view may be used here without any problems.
What about unordred_map, do it constexpr may be tricky, but there are not many values and constexpr flat map can be easialy created with standard algorithms
[100%] Linking CXX executable userver-samples-hello_service
/usr/bin/ld: невозможно найти -lHttp_Parser: Нет такого файла или каталога
collect2: ошибка: выполнение ld завершилось с кодом возврата 1
❯ pacman -Ql http-parser
http-parser /usr/
http-parser /usr/include/
http-parser /usr/include/http_parser.h
http-parser /usr/lib/
http-parser /usr/lib/libhttp_parser.so
http-parser /usr/lib/libhttp_parser.so.2.9
http-parser /usr/lib/libhttp_parser.so.2.9.4
http-parser /usr/share/
http-parser /usr/share/licenses/
http-parser /usr/share/licenses/http-parser/
http-parser /usr/share/licenses/http-parser/LICENSE-MIT
Apparently it's case-sensitive and package for arch linux comes with different case.
Manual replacement of -lHttp_Parser
with -lhttp_parser
in build files helped.
Please, pre-check Python version to prevent this error. Thank you.
I have compiled using Python 3.4:
File "userver/plugins/external_deps/impl/cmake_generator.py", line 81
result: dict = {}
SyntaxError: invalid syntax
CMake Error at CMakeLists.txt:70 (message):
Generating cmake files failed with exit code: 1
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.