bloomberg / blazingmq Goto Github PK
View Code? Open in Web Editor NEWA modern high-performance open source message queuing system
Home Page: https://bloomberg.github.io/blazingmq/
License: Apache License 2.0
A modern high-performance open source message queuing system
Home Page: https://bloomberg.github.io/blazingmq/
License: Apache License 2.0
No - that's just an idea of how to improve the performance of BlazingMQ possibly.
Recently I checked Profile-Guided Optimization (PGO) improvements on multiple projects. The results are here. I think these results are quite promising and can be a stimulus to test PGO with BlazingMQ.
We need to perform PGO benchmarks on BlazingMQ. And if it shows improvements (reduced CPU usage/reduced latency/anything else) - add a note to the documentation about possible improvements in BlazingMQ performance with PGO. Providing an easier way (e.g. a build option) to build scripts with PGO can be helpful for the end-users too.
No response
Compiling on Fedora 39 failed because of linking issue;
[100%] Linking CXX executable bmqtool.tsk
/usr/bin/ld: /usr/lib64/libfl.so: undefined reference to `yylex'
which could be traced to how flex 2.6.x changed the way yylex behaves.
No response
Env - OS: Fedora 39
Step - run bin/build-ubuntu.sh (commenting out the apt update and installations)
Expected - to compile/link without errors
Errors -
[100%] Linking CXX executable bmqtool.tsk
/usr/bin/ld: /usr/lib64/libfl.so: undefined reference to `yylex'
0.90.21
I solved the compilation issue by adding target_link_options(bmq PUBLIC "LINKER:-as-needed")
to the CMakeLists.txt in the src/groups/bmq/ folder,
if(NOT CMAKE_HOST_SYSTEM_NAME MATCHES "Darwin")
target_link_libraries(bmq PUBLIC "${FLEX_LIBRARIES}")
target_link_options(bmq PUBLIC "LINKER:-as-needed")
endif()
However I am not sure if that's the right solution without affecting how BlazingMQ would behave.
There are ~2900 -Wconversion
warnings during compilation (out of ~3000 warnings in total). All or most of them are unprobable to cause any problems in our runtime.
But there are concerns:
Info to read:
https://gcc.gnu.org/wiki/NewWconversion
Some of these conversion warnings are straightforward to fix. While the scale of 2900 warnings is huge, it might be good to split the work into a smaller chunks: make changes to the simplest cases, on a file or several-files basis, no more than 5-10 changes per PR etc...
Suppress -Wconversion
warning during compilation to remove noise - but for me it's better to fix it
main
branch buildBuild BlazingMQ
-Wconversion
to find an intersting place to fixProblem is for very long delayed queue.
Delayed message queue that can hold millions of messages for long enough (e.g. 365 days), and persistent is required.
With rabbitmq and delayed message queue plugin it's possible to achieve that partly, mostly because of limitations in mnesia.
The ability to send JSON via HTTP posts to a wehbook endpoint which then takes that JSON and adds it to the message queue.
I know MQ's + HTTP is kind of an antipatern and usually the advice is to write your own middleware. So far I have not seen a message queue that offers this feature, but it seems like a common enough problem that people should not have to write their own customer middleware every time.
An example of this is can be found trying to search for ways to integrate HTTP messages with RabbitMQ and finding this project https://github.com/dougwatson/http2amqp
The ideal scenario would be if you could create a webhook endpoint on BlazingMQ which would then be able to receive JSON messages from HTTP (push/pub) sources which would convert the JSON into a message and place it into a queue.
Custom middleware
No
According to the documentation, clients for Java and Python are available, such as the following:
https://github.com/bloomberg/blazingmq-sdk-java
I would be interested in helping contribute to an sdk for dotnet if the project maintainers feel this might be of use to the broader community.
No response
Currently we have 3 ways to throttle logging:
With mwcu::ThrottledActionParams
:
blazingmq/src/groups/mqb/mqbblp/mqbblp_cluster.cpp
Lines 414 to 419 in 5f65a2a
With bdlmt::Throttle
:
blazingmq/src/groups/mqb/mqbs/mqbs_filestore.cpp
Line 3669 in 5f65a2a
With BALL_LOGTHROTTLE
:
The good example of a problem: mqbblp::Cluster
, here we have many-many fields responsible for throttled logging in the header:
blazingmq/src/groups/mqb/mqbblp/mqbblp_cluster.h
Lines 254 to 292 in 5f65a2a
Removing these fields by using a local BALL_LOGTHROTTLE
will greatly simplify the class.
Replace field-stored mwcu::ThrottledActionParams
with local BALL_LOGTHROTTLE
where it's possible.
No response
docker compose is freezing see below when running. the machine i'm using is a ec2 machine: t2.micro
ubuntu@ip-172-31-85-234:~/development/blazingmq$ docker compose -f docker/single-node/do cker-compose.yaml up --build -d
[+] Running 1/1
✘ bmqtool Error 0.1s
[+] Building 92.6s (10/20)
=> [bmqbrkr internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 2.13kB 0.0s
=> [bmqbrkr internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [bmqbrkr internal] load metadata for docker.io/library/ubuntu:22.04 0.2s
=> CACHED [bmqbrkr builder 1/12] FROM docker.io/library/ubuntu:22.04@sha256:0bced47fffa3361afa981854fcabcd4577cd43cebbb808cea2b1f33a3dd7f508 0.0s
=> [bmqbrkr internal] load build context 0.2s
=> => transferring context: 74.84kB 0.2s
=> CACHED [bmqbrkr builder 2/12] RUN apt update && apt -y install ca-certificates 0.0s
=> CACHED [bmqbrkr builder 3/12] RUN apt install -y --no-install-recommends build-essential gdb curl python3.10 ninja-build pkg-config bison libfl-dev libbenchma 0.0s
=> CACHED [bmqbrkr builder 4/12] RUN mkdir -p /deps_build/CMake && cd /deps_build/CMake && curl -s -L -O https://github.com/Kitware/CMake/releases/download/v3.27.1/cmake-3.27.1-linux-x86_64 0.0s
=> CACHED [bmqbrkr builder 5/12] WORKDIR /workspace 0.0s
=> CACHED [bmqbrkr builder 6/12] COPY docker/build_deps.sh bin/build_deps.sh 0.0s
=> [bmqbrkr builder 7/12] RUN bin/build_deps.sh 92.1s
=> => # [487/1488] Generating /workspace/srcs/bde/groups/bsl/bslma/bslma_sharedptrinplacerep_cpp03.h
=> => # [488/1488] Generating /workspace/srcs/bde/groups/bsl/bslma/bslma_stdallocator_cpp03.h
=> => # [489/1488] Generating /workspace/srcs/bde/groups/bsl/bslma/bslma_allocatortraits_cpp03.h
=> => # [490/1488] Generating /workspace/srcs/bde/groups/bsl/bslma/bslma_managedptr_cpp03.h
=> => # [491/1488] Generating /workspace/srcs/bde/groups/bsl/bslma/bslma_constructionutil_cpp03.h
=> => # [492/1488] Building CXX object groups/bsl/CMakeFiles/bslh-iface.dir/bslh/bslh_fibonaccibadhashwrapper.cpp.o
container should be able to be created
1.0
No response
Successfully compiled the library, and now trying to compile the tutorial source code.
I successfully compiled the library using the supplied sh script. Now how can I install the libraries in usual places (/usr/local/*) for local development?
No response
I read the docs.
There seems blazingmq to be no support for adding machines to improve overall scalability like kafka(add more machines, increase partitions).
In the Compression section of the documentation, the Summary section states:
Producer applications can indicate to the BlazingMQ SDK to compress a message. BlazingMQ SDK will compress the message (both payload and properties) before sending it to the BlazingMQ back-end.
Later on the same page in the Important Notes section, there is an apparent contradiction:
Only the message payload is compressed. Message properties are not compressed in order to ensure that the BlazingMQ message broker can read the properties efficiently if needed (for example, properties are read by the broker for evaluating subscriptions).
I'm assuming both of these can't be correct — properties cannot be both compressed and not compressed. (I mean they could be, but it would seem surprising)
The documentation is self-consistent.
1.0.0
No response
BlazingMQ a high-performance, reliable, and feature-rich message queueing system, has been released.
BlazingMQ provides a number of features that make it a powerful and versatile message queueing system, including:
Durability: Messages are persisted to disk, so they are not lost even if a broker or client fails.
Fault tolerance: BlazingMQ can tolerate the failure of individual brokers or clients without affecting the overall system.
High performance: BlazingMQ can handle millions of messages per second, making it suitable for high-volume applications.
High availability: BlazingMQ can be configured to be highly available, so that messages are always delivered even if some of the brokers are unavailable.
Rich feature set: BlazingMQ provides a number of features that are not available in other message queueing systems, such as support for various message routing strategies, compression, strong consistency, and poison pill detection.
BlazingMQ is a good choice for a variety of applications, including:
Real-time streaming applications
Distributed computing applications
Microservices applications
Event-driven applications
The release of BlazingMQ includes a number of new features and improvements, including:
Support for multiple message routing strategies
Compression support
Strong consistency support
Poison pill detection
Improved performance and scalability
The new ShellCheck linter job fails on three existing shell scripts in our repository:
./bin/build-ubuntu.sh
,./bin/build-darwin.sh
, and./src/applications/bmqbrkr/run
.CI checks should not fail.
https://github.com/bloomberg/blazingmq/actions/runs/5755158855/job/15602005338
HEAD
No response
The docs/README.md
file describes how to do local development of our Just the Docs documentation website on a Mac system, but has some stray references to tools that most users will not have access to.
Anyone should be able to follow the steps for local Mac development on our Just the Docs documentation, so they can easily submit patches.
.
HEAD
No response
The expect_XYZ
functions in bmqa_mocksession
(e.g., expect_configureQueueSync
) are run using the BMQA_EXPECT_CALL
macro from a mock, and are meant to parallel the functions XYZ
(e.g., configureQueueSync
) in bmqa_abstractsession
. At the moment, several of these functions expect_XYZ
differ from their corresponding XYZ
functions in the const
ness of their parameters:
expect_closeQueueSync(QueueId*, ...)
, but closeQueueSync(const QueueId*, ...)
expect_closeQueueAsync(QueueId*, ...)
, but closeQueueAsync(const QueueId*, ...)
expect_configureQueueSync(QueueId*, ...)
, but configureQueueSync(const QueueId*, ...)
As a result, it is not possible to use BMQA_EXPECT_CALL
from a mock session that implements the AbstractSession
interface.
The arguments to expect_XYZ
functions should match the arguments to XYZ
functions in their const
ness.
.
HEAD
No response
We build systems on cloud with C++, Java, Go and Python languages.
Although Go and Python can interactive whith C++ SDK, but it would be nice if there is native support.
No response
The code is using internal bloomberg tools to find and build using the .mem and .dep definitions
that means that common tools that relies in cmake find_package to show and visualize cmake information will not be able to run with this project.
would it be ok to use the (documented) way to develop with bde, and add the find_package(bsl)
int he places that are needed? This way we can package blazingmq together with the packaged version of bde-tools
and bde
.
Use standard cmake for opensource builds
No response
Both of these constants happen to have the value 0
, but in principle we should be assigning the default subqueue ID to a subqueue id variable, rather than the default subscription id to a subqueue id variable.
No
I would like to have a library for using blazingMQ with NodeJS
No response
Currently, we have two supported ways of building BlazingMQ:
bin/
for Ubuntu and Darwin. These build scripts are very useful for managing some of the complexity of building our two biggest dependencies, bde
and ntf-core
, but for a user to build with them, they need to read through the build script first to ensure that they have the proper package manager dependencies installed. They also need to read through the script to see what environment variables they can pass to control the building and installation of BlazingMQ and its dependencies.Additionally, we may soon have vcpkg support coming. This will not make the above obsolete for all users, especially if they want to do development on the broker itself.
Other open source projects (for example, OBS, Kubernetes, or GNU/Emacs) maintain documentation that walks users through all supported ways of building the software. This is useful in addition to the above build scripts; Kubernetes, for instance, uses its build documentation to describe how to use the build scirpts.
To lower the cost to entry for people wishing to contribute to the project, it would be beneficial to include better documentation on how to build and install a BlazingMQ broker and client library. While this is especially important with regard to the broker, it still is beneficial for users who would prefer building themselves, perhaps if they are packaging BlazingMQ or are in an environment where they aren't able to use our potential vcpkg support.
No response
No response
No response
However, I can see one very unprobable scenario with a possible bug here.
Within UriParse::initialize
we increment int s_initialized
counter in a thread-safely manner (within the locked mutex).
blazingmq/src/groups/bmq/bmqt/bmqt_uri.cpp
Line 195 in 342b608
We early return if this counter is greater than 1, meaning, that some other code has already incremented it and initialized a shared state which is const-read-thread-safe.
However, we do bmqt::UriParse::initialize
every time we call a constructor for bmqt::Uri
, and do not deinitialize bmqt::UriParser
until we call a destructor bmqt::Uri::~Uri
.
This means, that if we are able to keep approx 2^31 bmqt::Uri
s in memory, we'll get int
overflow.
5a. Because of this possible overflow, the condition for early exit will fail on a new bmqt::Uri
construction, and we might re-initialize some const-read-thread-safe RegEx
, when some other thread might use it
5b. It is also possible to just launch bmqt::UriParser::initialize
2.2kkk times to get overflow. Even if we do call bmqt::UriParser::shutdown
the same number of times afterwards (according to contract), there will be a problem.
0.90.21
There are 2 possible solutions.
The minimal change (and safe) solution is to change the counter type int
-> bsls::Types::Int64
, so it will be impossible to reach overflow.
The good solution is to give up on a custom reference counting of bmqt::UriParser::initialize
and initialize it once and for all as a singleton, using the appropriate tool to initialize: BSLMT_ONCE_DO
https://bloomberg.github.io/bde-resources/doxygen/bde_api_prod/group__bslmt__once.html
The greatest benefit of this approach is that we don't lock the mutex every time we need to build a bmqt::Uri
. As you can see here (https://github.com/bloomberg/bde/blob/88f20d8057d462e2bdaf27b757ff3bf335555dcd/groups/bsl/bslmt/bslmt_once.cpp#L25), bslmt::Once
can check the need to do something without locking the internal mutex, when the state of once was already decided. As a result, we'll be able to work with bmqt::Uri
s much, much faster from different threads.
The difference: we'll not free these resources gracefully to make the allocator happy in the end of the program. I don't think this difference is bad, because the needed allocated memory is small, we already have some singletons allocated like this, and the runtime performance gain is huge.
We can also deprecate bmqt::UriParser::initialize
, bmqt::UriParser::shutdown
or make them no-op.
Our tests are currently using older versions of certain GitHub Actions which rely on the outdated Node.js version 16:
Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/checkout@v3, actions/cache@v3. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
This does not cause the tests to fail but adds noisy warnings to Actions output.
No response
This can be seen on any run of the tests
workflow, as in: https://github.com/bloomberg/blazingmq/actions/runs/7805114652
HEAD
These warnings can be fixed by updating to actions/checkout@v4
and actions/cache@v4
, which do not have any breaking changes over our current versions. A similar change has already been done in the Python SDK: bloomberg/blazingmq-sdk-python@4944dcf
There is currently a single useful operator for string properties (==
and !=
). For use-cases that need more powerful pattern matching, the following operations would be useful:
/var/log/foobar/*/*.log.
)A regex operator would solve many of those problems. We can use the following library to compile regexes without introducing additional dependencies to the broker:
https://github.com/bloomberg/bde/blob/d6674e1df52b62d66942ca318882f225d26104bf/groups/bdl/bdlpcre/bdlpcre_regex.h#L669
No response
We don't have automatic checks for M1/M2 builds.
According to this roadmap, Apple Silicon (M1) is in public beta in GitHub Actions:
Might be possible to use the corresponding runner to check darwin build script automatically.
No response
The expression language currently does not provide a clear way to check for the existence of a property.
Let's assume I have a queue with two types of messages.
Messages of type A have the following properties:
radius = 34
color = "blue"
Messages of type B have the following properties:
height: 53
width: 100
color = "red"
If I want to filter for messages that contain a radius
field in a subscription, I can currently recur to the following workaround expressions:
radius == radius
radius >= 0 || radius < 0
however my intent to check for existence is not clear.
I would love if the expression language allowed me to check for existence of fields explicitly. This can be done by either checking if a field:
!= NULL
for the existence check.
and
== NULL
for the missing property check.
Alternatively an operator for exists() can be introduced. This alternative would require a unary negation operator.
No response
Place where the problem might occur:
This static local variable initialization is not thread safe in general. The probability to observe a bug is small though.
Here are some details and advices:
https://bloomberg.github.io/bde-resources/doxygen/bde_api_prod/group__bslmt__once.html#_details
Here is how we did a safe initialization for another component:
blazingmq/src/groups/bmq/bmqt/bmqt_correlationid.cpp
Lines 33 to 42 in 08a6d16
Will be good to do the same for the problematic line of code above.
Try to build multiple SubscriptionHandle
s at the same time from different threads just after the app starts.
0.90.18
No response
This feature is not supported in non-developer releases, and its use cases have been mostly superseded by BlazingMQ subscriptions.
Hi there, first time hearing about this project,
I am used to zmq and zeroless to either send heavy data msgs or bulk async request, and I have easy python wrappers to interface with
Can you please explain how is is blazingmq better, in terms of benchmarks performance, features, and other metrics ?
Excited to experiment with it
Thanks a lot and all the best!
When running the initial docker compose
command on an M2 silicon MacBook Pro, the build fails with the following error:
36.19 rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2
Docker compose completes successfully with no errors.
docker compose -f docker/single-node/docker-compose.yaml up --build -d
Here is the error in context:
FROM --platform=linux/amd64 docker.io/ubuntu:22.04 AS builder
36.16 ++ BDE_CMAKE_UFID=opt_64_cpp17
36.16 ++ export BDE_CMAKE_BUILD_DIR=_build/unix-linux-aarch64-6.4.16-gcc-11.4.0-opt_64_cpp17
36.16 ++ BDE_CMAKE_BUILD_DIR=_build/unix-linux-aarch64-6.4.16-gcc-11.4.0-opt_64_cpp17
36.16 ++ export CC=/usr/bin/gcc
36.16 ++ CC=/usr/bin/gcc
36.16 ++ export CXX=/usr/bin/g++
36.16 ++ CXX=/usr/bin/g++
36.16 ++ export BDE_CMAKE_TOOLCHAIN=/workspace/srcs/bde-tools/BdeBuildSystem/toolchains/linux/gcc-default.cmake
36.16 ++ BDE_CMAKE_TOOLCHAIN=/workspace/srcs/bde-tools/BdeBuildSystem/toolchains/linux/gcc-default.cmake
36.16 ++ export BDE_CMAKE_INSTALL_DIR=/workspace/_install
36.16 ++ BDE_CMAKE_INSTALL_DIR=/workspace/_install
36.16 + build
36.16 + build_bde
36.16 + pushd srcs/bde
36.16 /workspace/srcs/bde /workspace
36.16 + bbs_build configure
36.19 rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2
36.19 Error: Command '['cmake', '/workspace/srcs/bde', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=Release', '-DBDE_BUILD_TARGET_64=1', '-DBDE_BUILD_TARGET_CPP17=1', '-DBdeBuildSystem_DIR:PATH=/workspace/srcs/bde-tools/BdeBuildSystem', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DBBS_BUILD_SYSTEM=ON', '-DBBS_USE_WAFSTYLEOUT=OFF', '-DBBS_CPP11_VERIFY_NO_CHANGE=OFF', '-DCMAKE_INSTALL_PREFIX=/opt/bb', '-DCMAKE_INSTALL_LIBDIR=lib64', '-DCMAKE_TOOLCHAIN_FILE=/workspace/srcs/bde-tools/BdeBuildSystem/toolchains/linux/gcc-default.cmake', '-DCMAKE_PREFIX_PATH:PATH=///opt/bb', '--log-level=ERROR']' died with <Signals.SIGTRAP: 5>.
36.19 Configuration cmd:
36.19 cmake /workspace/srcs/bde -G Ninja -DCMAKE_BUILD_TYPE=Release -DBDE_BUILD_TARGET_64=1 -DBDE_BUILD_TARGET_CPP17=1 -DBdeBuildSystem_DIR:PATH=/workspace/srcs/bde-tools/BdeBuildSystem -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBBS_BUILD_SYSTEM=ON -DBBS_USE_WAFSTYLEOUT=OFF -DBBS_CPP11_VERIFY_NO_CHANGE=OFF -DCMAKE_INSTALL_PREFIX=/opt/bb -DCMAKE_INSTALL_LIBDIR=lib64 -DCMAKE_TOOLCHAIN_FILE=/workspace/srcs/bde-tools/BdeBuildSystem/toolchains/linux/gcc-default.cmake -DCMAKE_PREFIX_PATH:PATH=///opt/bb --log-level=ERROR
The sample consumer with the problem exposed:
https://github.com/bloomberg/blazingmq/pull/132/files
The crash log looks like this:
20OCT2023_13:24:39.771 (6108901376) FATAL mqbblp_queuehandle.cpp:634 Assertion failed: subscription->d_downstreamSubQueueId == downstreamSubId, stack trace:
(0): bmqAssertHandler(char const*, char const*, int)+0xe4 at 0x104461d84 in bmqbrkr.tsk
(1): BloombergLP::bsls::Assert::failOnViolation(BloombergLP::bsls::AssertViolation const&)+0x54 at 0x1053f82f4 in bmqbrkr.tsk
(2): BloombergLP::bsls::Assert::invokeHandler(BloombergLP::bsls::AssertViolation const&)+0x24 at 0x1053f857c in bmqbrkr.tsk
(3): BloombergLP::mqbblp::QueueHandle::registerSubscription(unsigned int, unsigned int, BloombergLP::bmqp_ctrlmsg::ConsumerInfo const&, unsigned int)+0x324 at 0x1048691a8 in bmqbrkr.tsk
(4): BloombergLP::mqbblp::Routers::Consumer::registerSubscriptions(BloombergLP::mqbi::QueueHandle*)+0xb8 at 0x10496acfc in bmqbrkr.tsk
(5): BloombergLP::mqbblp::Routers::AppContext::registerSubscriptions()+0x80 at 0x10496e640 in bmqbrkr.tsk
(6): BloombergLP::mqbblp::RootQueueEngine::configureHandle(BloombergLP::mqbi::QueueHandle*, BloombergLP::bmqp_ctrlmsg::StreamParameters const&, bsl::function<void (BloombergLP::bmqp_ctrlmsg::Status const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&)> const&)+0xe3c at 0x10494bd24 in bmqbrkr.tsk
(7): BloombergLP::mqbblp::LocalQueue::configureHandle(BloombergLP::mqbi::QueueHandle*, BloombergLP::bmqp_ctrlmsg::StreamParameters const&, bsl::function<void (BloombergLP::bmqp_ctrlmsg::Status const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&)> const&)+0xf8 at 0x1047e8e14 in bmqbrkr.tsk
(8): BloombergLP::mqbblp::Queue::configureHandle(BloombergLP::mqbi::QueueHandle*, BloombergLP::bmqp_ctrlmsg::StreamParameters const&, bsl::function<void (BloombergLP::bmqp_ctrlmsg::Status const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&)> const&)+0xb8 at 0x10480291c in bmqbrkr.tsk
(9): BloombergLP::mqbblp::QueueHandle::configureDispatched(BloombergLP::bmqp_ctrlmsg::StreamParameters const&, bsl::function<void (BloombergLP::bmqp_ctrlmsg::Status const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&)> const&)+0x344 at 0x10486bc00 in bmqbrkr.tsk
Will be good to handle this in the broker, so other SDKs (including custom-made) are not able to cause crash. Also will be good to add check for this in the SDK, so the troublesome message is not sent at all.
QueueOptions
with subscriptionsQueueOptions
for different fanout appIds.0.90.18
No response
Files: m_bmqtool_inpututil.h
and m_bmqtool_inpututil.cpp
There are functions with arguments passed as a pointer, with assumption that these pointer values are not null.
Example:
blazingmq/src/applications/bmqtool/m_bmqtool_inpututil.cpp
Lines 50 to 53 in 9b692fe
Add checks to pointer arguments to functions in m_bmqtool_inpututil
.
Example:
blazingmq/src/groups/mqb/mqbmock/mqbmock_cluster.cpp
Lines 380 to 384 in 9b692fe
No response
When performing the following search in the Blazing MQ documentation: https://bloomberg.github.io/blazingmq/docs/apidocs/python_apidocs/search.html?q=thread&check_keywords=yes&area=default
The synopsis for each match is a file not found error message. But the link to the document works. Below is sample text from the search results.
Search Results
Search finished, found 3 page(s) matching the search query.
Examples
...Page not foundThe page you requested could not be found. Try using the navigation or search to find what you're looking for or go to this site's home page.Back to topCopyright © 2023 Bloombergblazingmq
...Page not foundThe page you requested could not be found. Try using the navigation or search to find what you're looking for or go to this site's home page.Back to topCopyright © 2023 Bloomberg
To only display the error if there's an actual error.
This should be reproducible with the link above.
HEAD
This issue was reported in the Python SDK as bloomberg/blazingmq-sdk-python#11. The Python SDK documentation is hosted in this repository, so fixing it will involve a code change to the Python SDK codebase along with a PR here.
Ultimately it will make more sense to host the Python API docs on GH pages in the Python SDK repo.
Timeout under 1us (for example, 100ns) is interpreted as infinite timeout
Only zero timeout should be interpreted as "infinite"
Create a sync session
Pass 1ns as the timeout to Session::nextEvent
The call will block instead of returning immediately
1.13.3
No response
Currently, we have 2 different states for an existing queue: active and inactive.
The difference between these 2 queues is that active queues have both storage level representation and runtime object representation mqbi::Queue
, while inactive queue has only storage level representation.
The current implementation of PURGE QUEUE / PURGE DOMAIN iterates through mqbi::Queue
objects so it is not aware of inactive queues at all. The implementation is not able to purge inactive queues.
The implementation should work on a storage level instead, where we have all the information needed to purge both active and inactive queues.
0.90.20
No response
C API is very useful in general, and can become a bridge for API in other languages as well.
Since implementing a C API from ground-up would be too much effort, we plan to write a C wrapper over the existing C++ API.
No response
Build script(both build_ubuntu.sh and build_darwin.sh) fail to build blazingmq. Build produces a warning and fails to continue build due to flag set to fail on warnings. The following error occurs:
In file included from /home/jon/cs/bb/simon-fork/blazingmq/include/bslstl_ownerless.h:72, from /home/jon/cs/bb/simon-fork/blazingmq/include/bsl_memory.h:152, from /home/jon/cs/bb/simon-fork/blazingmq/include/bdlf_bind.h:900, from /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntccfg/ntccfg_bind.h:40, from /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntccfg/ntccfg_platform.h:22, from /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.h:22, from /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp:16: In destructor ‘bsl::shared_ptr<ELEMENT_TYPE>::~shared_ptr() [with ELEMENT_TYPE = BloombergLP::ntci::Executor]’, inlined from ‘BloombergLP::ntcs::ObserverRef<TYPE>::~ObserverRef() [with TYPE = BloombergLP::ntci::Executor]’ at /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_observer.h:533:32, inlined from ‘virtual void BloombergLP::ntcs::Strand::moveAndExecute(BloombergLP::ntci::Executor::FunctorSequence*, const BloombergLP::ntci::Executor::Functor&)’ at /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp:263:5: /home/jon/cs/bb/simon-fork/blazingmq/include/bslstl_sharedptr.h:5045:28: error: ‘*(bsl::shared_ptr<BloombergLP::ntci::Executor>*)((char*)&executorRef + offsetof(BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>,BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>::d_shared)).bsl::shared_ptr<BloombergLP::ntci::Executor>::d_rep_p’ may be used uninitialized [-Werror=maybe-uninitialized] 5045 | d_rep_p->releaseRef(); | ~~~~~~~~~~~~~~~~~~~^~ /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp: In member function ‘virtual void BloombergLP::ntcs::Strand::moveAndExecute(BloombergLP::ntci::Executor::FunctorSequence*, const BloombergLP::ntci::Executor::Functor&)’: /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp:254:43: note: ‘*(bsl::shared_ptr<BloombergLP::ntci::Executor>*)((char*)&executorRef + offsetof(BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>,BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>::d_shared)).bsl::shared_ptr<BloombergLP::ntci::Executor>::d_rep_p’ was declared here 254 | ntcs::ObserverRef<ntci::Executor> executorRef(&d_executor); | ^~~~~~~~~~~ In destructor ‘bsl::shared_ptr<ELEMENT_TYPE>::~shared_ptr() [with ELEMENT_TYPE = BloombergLP::ntci::Executor]’, inlined from ‘BloombergLP::ntcs::ObserverRef<TYPE>::~ObserverRef() [with TYPE = BloombergLP::ntci::Executor]’ at /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_observer.h:533:32, inlined from ‘virtual void BloombergLP::ntcs::Strand::execute(const BloombergLP::ntci::Executor::Functor&)’ at /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp:230:5: /home/jon/cs/bb/simon-fork/blazingmq/include/bslstl_sharedptr.h:5045:28: error: ‘*(bsl::shared_ptr<BloombergLP::ntci::Executor>*)((char*)&executorRef + offsetof(BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>,BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>::d_shared)).bsl::shared_ptr<BloombergLP::ntci::Executor>::d_rep_p’ may be used uninitialized [-Werror=maybe-uninitialized] 5045 | d_rep_p->releaseRef(); | ~~~~~~~~~~~~~~~~~~~^~ /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp: In member function ‘virtual void BloombergLP::ntcs::Strand::execute(const BloombergLP::ntci::Executor::Functor&)’: /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp:221:43: note: ‘*(bsl::shared_ptr<BloombergLP::ntci::Executor>*)((char*)&executorRef + offsetof(BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>,BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>::d_shared)).bsl::shared_ptr<BloombergLP::ntci::Executor>::d_rep_p’ was declared here 221 | ntcs::ObserverRef<ntci::Executor> executorRef(&d_executor); | ^~~~~~~~~~~ cc1plus: all warnings being treated as errors gmake[3]: *** [CMakeFiles/ntcs.dir/build.make:594: CMakeFiles/ntcs.dir/groups/ntc/ntcs/ntcs_strand.cpp.o] Error 1 gmake[3]: *** Waiting for unfinished jobs.... gmake[2]: *** [CMakeFiles/Makefile2:7198: CMakeFiles/ntcs.dir/all] Error 2 gmake[1]: *** [Makefile:166: all] Error 2
This results in the following failure:
CMake Error at /home/jon/cs/bb/simon-fork/blazingmq/build/ntf/Linux/opt_dbg_exc_mt_64/cmake_install.cmake:1384 (file): file INSTALL cannot find "/home/jon/cs/bb/simon-fork/blazingmq/build/ntf/Linux/opt_dbg_exc_mt_64/lib/libntc.a": No such file or directory.
No response
./bin/build-darwin.sh
or ./bin/build-ubuntu.sh
1.0.0
Issue was encountered on a intel macbook as well as a machine running the Fedora linux distro
I'm attempting to start the single-node test project but when I run docker-compose
I get the following:
docker compose -f docker/single-node/docker-compose.yaml up --build -d
[+] Running 0/1
⠿ bmqtool Error
Error response from daemon: pull access denied for bmqbrkr, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
I expected the docker container(s) to start up.
> git clone [email protected]:bloomberg/blazingmq.git
> cd blazingmq
> docker login
> docker compose -f docker/single-node/docker-compose.yaml up --build -d
0.0.0
I checked out docker hub but I don't see the image bmqbrkr
anywhere.
Currently our GitHub actions lack a linting step for shell scripts. Doing so might have been able to catch some of the comments of #34 and #54 before review.
Proposed solution is to add a step similar to the clang-format
step we already run on C++ code.
No response
Regarding your
https://bloomberg.github.io/blazingmq/docs/performance/benchmarks
its a good report and thanks for publishing it and having following quesitons.
Our current API documentation was generated using bde-tools
's bdedox
executable. Since our comments are no longer in BDE-style, running doxygen
on the codebase does not pick up several things our current published API docs have, like package group, package, and component-level documentation, some BDE formatting, and \brief
descriptions automatically generated from the first sentence of a documentation comment.
Additionally, some BDE-style documentation failed to convert, as in the bmqa_abstractsession
component.
Running doxygen
on the codebase should produce up-to-date documentation, including all our package group, package, and component documentation, and should support Doxygen-style annotations, such as \deprecated
and \par
.
doxygen
on the codebase.HEAD
No response
Hi,
I am new here, trying to build the code on Ubuntu 20.04. I ran into an issue running cmake:
CMake Error at /usr/local/share/cmake/BdeBuildSystem/BdeEmitPkgConfigFile.cmake:17 (message):
You need to supply LIBDIR to bbs_emit_pc_files or set CMAKE_INSTALL_LIBDIR
Call Stack (most recent call first):
etc/cmake/TargetBMQStyleUor.cmake:119 (bbs_emit_pkgconfig_file)
src/groups/mqb/CMakeLists.txt:27 (bmq_emit_pkg_config)
I am not an expert user of CMake, can anybody tell me what I am missing?
I want to implement a client to your broker.
Will there be a protocol specification added?
No response
There are several places where C-style conversions are used.
Search the codebase for (double)
:
mwcst_printutil.cpp - 2
mwcst_statutil.cpp - 3
mwcu_printutil.t.cpp - 3
It's possible to find some conversions by searching for (int)
too, but this search shows more false alarms and unrelated (int)
usage. There are still conversions to be fixed, like:
int BasicTableInfoProvider::numColumns(int level) const
{
if (level == 0) {
return (int)d_columns.size();
}
else {
return (int)d_columnGroups.size();
}
}
Another possible searches: int(
, double(
(with a space in the beginning)
Use static_cast
Warning: there might be some auto-generated files with C-style conversions, usually with a name ending with messages.cpp
or messages.h
, don't need to modify it.
No response
The website says that Blazingmq was developed entirely from the ground up without the use of any external library.
but that's not true, as soon as you open:
blazingmq/src/groups/mwc/grou[/mwc.dep:
you see that it's using BDE:
bal bdl bsl
No response
2nd paragraph of the website states that there's not a single external library. But it uses BDE.
1.0.0
No response
cppcheck
output:
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:945:9: error: Non-local variable 'g_bufferFactory_p' will use pointer to local variable 'bufferFactory'. [danglingLifetime]
g_bufferFactory_p = &bufferFactory;
^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:945:33: note: Address of variable taken here.
g_bufferFactory_p = &bufferFactory;
^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:941:40: note: Variable created here.
bdlbb::PooledBlobBufferFactory bufferFactory(k_LONG_ENTRY_LENGTH * 2,
^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:945:9: note: Non-local variable 'g_bufferFactory_p' will use pointer to local variable 'bufferFactory'.
g_bufferFactory_p = &bufferFactory;
^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:946:9: error: Non-local variable 'g_miniBufferFactory_p' will use pointer to local variable 'miniBufferFactory'. [danglingLifetime]
g_miniBufferFactory_p = &miniBufferFactory;
^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:946:33: note: Address of variable taken here.
g_miniBufferFactory_p = &miniBufferFactory;
^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:943:40: note: Variable created here.
bdlbb::PooledBlobBufferFactory miniBufferFactory(k_ENTRY_LENGTH,
^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:946:9: note: Non-local variable 'g_miniBufferFactory_p' will use pointer to local variable 'miniBufferFactory'.
g_miniBufferFactory_p = &miniBufferFactory;
Will be nice to remove global objects construction from here:
blazingmq/src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp
Lines 941 to 946 in ca6491f
And make it a field of the Tester
class, which is constructed locally for each test.
Current implementation has a good point - it builds these global objects once and saves some processing time on rebuilding it. But this also might be a problem, because it's a shared state among tests.
No response
Having trouble building the image on - mac os venture Apple m2 chip
On running command - docker compose -f docker/single-node/docker-compose.yaml up --build -d
the script build_deps.sh in the container is failing to execute complaining
gcc: error: unrecognized command-line option ‘-m64’
ninja: build stopped: subcommand failed.
Attached screenshot for reference.
No response
running command -> docker compose -f docker/single-node/docker-compose.yaml up --build -d
1
If you search for ExecUtil
usage, you can find some files with unused #include <mwcsys_executil.h>
:
https://github.com/search?q=repo%3Abloomberg%2Fblazingmq%20ExecUtil&type=code
Will be good to remove these unused includes where possible, while keeping files mwcsys_executil.h
, mwcsys_executil.cpp
, mwcsys_executil.t.cpp
. These files might be used by existing library's clients.
No response
On running:
$ docker compose -f docker/single-node/docker-compose.yaml up --build -d
Getting the following error:
5.847 + bbs_build configure
5.847 /workspace/srcs/bde /workspace
5.873 qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory
5.873 Error: Command '['cmake', '/workspace/srcs/bde', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=Release', '-DBDE_BUILD_TARGET_64=1', '-DBDE_BUILD_TARGET_CPP17=1', '-DBdeBuildSystem_DIR:PATH=/workspace/srcs/bde-tools/BdeBuildSystem', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DBBS_BUILD_SYSTEM=ON', '-DBBS_USE_WAFSTYLEOUT=OFF', '-DBBS_CPP11_VERIFY_NO_CHANGE=OFF', '-DCMAKE_INSTALL_PREFIX=/opt/bb', '-DCMAKE_INSTALL_LIBDIR=lib64', '-DCMAKE_TOOLCHAIN_FILE=/workspace/srcs/bde-tools/BdeBuildSystem/toolchains/linux/gcc-default.cmake', '-DCMAKE_PREFIX_PATH:PATH=///opt/bb', '--log-level=ERROR']' returned non-zero exit status 255.
5.873 Configuration cmd:
5.873 cmake /workspace/srcs/bde -G Ninja -DCMAKE_BUILD_TYPE=Release -DBDE_BUILD_TARGET_64=1 -DBDE_BUILD_TARGET_CPP17=1 -DBdeBuildSystem_DIR:PATH=/workspace/srcs/bde-tools/BdeBuildSystem -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBBS_BUILD_SYSTEM=ON -DBBS_USE_WAFSTYLEOUT=OFF -DBBS_CPP11_VERIFY_NO_CHANGE=OFF -DCMAKE_INSTALL_PREFIX=/opt/bb -DCMAKE_INSTALL_LIBDIR=lib64 -DCMAKE_TOOLCHAIN_FILE=/workspace/srcs/bde-tools/BdeBuildSystem/toolchains/linux/gcc-default.cmake -DCMAKE_PREFIX_PATH:PATH=///opt/bb --log-level=ERROR
------
failed to solve: process "/bin/sh -c bin/build_deps.sh" did not complete successfully: exit code: 1
Laptop spec:
Docker Version: 24.0.6
Macbook M2 Air
Build to complete without any issues
1.13.3
N/A
Currently, AdminClient
is not suitable to use in with
statements ensuring auto freeing of resources.
Implement AbstractContextManager
interfaces in AdminClient
No response
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.