bloomberg/blazingmq

A modern high-performance open source message queuing system

---

BlazingMQ - A Modern, High-Performance Message Queue

BlazingMQ is an open source distributed message queueing framework, which focuses on efficiency, reliability, and a rich feature set for modern-day workflows.

At its core, BlazingMQ provides durable, fault-tolerant, highly performant, and highly available queues, along with features like various message routing strategies (e.g., work queues, priority, fan-out, broadcast, etc.), compression, strong consistency, poison pill detection, etc.

Message queues generally provide a loosely-coupled, asynchronous communication channel ("queue") between application services (producers and consumers) that send messages to one another. You can think about it like a mailbox for communication between application programs, where 'producer' drops a message in a mailbox and 'consumer' picks it up at its own leisure. Messages placed into the queue are stored until the recipient retrieves and processes them. In other words, producer and consumer applications can temporally and spatially isolate themselves from each other by using a message queue to facilitate communication.

BlazingMQ's back-end (message brokers) has been implemented in C++, and client libraries are available in C++, Java, and Python.

BlazingMQ is an actively developed project and has been battle-tested in production at Bloomberg for 8+ years.

This repository contains BlazingMQ message broker, BlazingMQ C++ client library and a BlazingMQ command line tool, while BlazingMQ Java client library can be found in this repository.

---

Menu

• Documentation
• Quick Start
• Building
• Installation
• Contributions
• License
• Code of Conduct
• Security Vulnerability Reporting

---

Documentation

Comprehensive documentation about BlazingMQ can be found here.

---

Quick Start

This article guides readers to build, install, and experiment with BlazingMQ locally in a Docker container.

In the companion article, readers can learn about some intermediate and advanced features of BlazingMQ and see them in action.

---

Building

bin/build-ubuntu.sh and bin/build-darwin.sh build BlazingMQ and its dependencies, respectively, on Ubuntu 22.04.2 LTS and Darwin 22.6.0. They can serve as a basis to build BlazingMQ on other systems.

To build BlazingMQ with plugins, pass '--plugins' argument with desired plugin names to the build script, e.g.

bin/build-ubuntu.sh --plugins plugin-1-name,plugin-2-name

With vcpkg

There is also support for building BlazingMQ with vpckg.

Before attempting to build, you will have to acquire flex, bison, and bde-tools for your system, as vcpkg cannot fetch them. Both flex and bison can likely be installed through your system's package manager. Clone bde-tools, we'll assume blazingmq/thirdparty/bde-tools for this guide.

Once the prerequisite tools are installed, you should be able to build BlazingMQ with the following:

export VCPKG_ROOT=/path/to/vcpkg
cmake --preset [preset-name] -DCMAKE_PREFIX_PATH=/path/to/thirdparty/bde-tools
cmake --build cmake.bld

For a list of presets, please look at the *-vcpkg configurations in CMakePresets.json.

---

Installation

This article describes the steps for installing a BlazingMQ cluster in a set of Docker containers, along with a recommended set of configurations.

---

Contributions

We welcome your contributions to help us improve and extend this project!

We welcome issue reports here; be sure to choose the proper issue template for your issue, so that we can be sure you're providing us with the necessary information.

Before sending a Pull Request, please make sure you have read our Contribution Guidelines.

---

License

BlazingMQ is Apache 2.0 licensed, as found in the LICENSE file.

---

Code of Conduct

This project has adopted a Code of Conduct. If you have any concerns about the Code, or behavior which you have experienced in the project, please contact us at [email protected].

---

Security Vulnerability Reporting

If you believe you have identified a security vulnerability in this project, please send an email to the project team at [email protected], detailing the suspected issue and any methods you've found to reproduce it.

Please do NOT open an issue in the GitHub repository, as we'd prefer to keep vulnerability reports private until we've had an opportunity to review and address them.

---

blazingmq's Issues

Fix `-Wconversion` warnings during compilation

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

There are ~2900 -Wconversion warnings during compilation (out of ~3000 warnings in total). All or most of them are unprobable to cause any problems in our runtime.

But there are concerns:

• Really interesting warnings could be lost in this noise.
• Logs become larger in general.
• Prevents from zero-warning compilation.

Info to read:
https://gcc.gnu.org/wiki/NewWconversion

Describe the solution you'd like

Some of these conversion warnings are straightforward to fix. While the scale of 2900 warnings is huge, it might be good to split the work into a smaller chunks: make changes to the simplest cases, on a file or several-files basis, no more than 5-10 changes per PR etc...

Alternatives you considered

Suppress -Wconversion warning during compilation to remove noise - but for me it's better to fix it

Contribution workflow

1. Go to the Github Actions for this project: https://github.com/bloomberg/blazingmq/actions
2. Find the nearest main branch build
3. Open logs for Build BlazingMQ
4. Ctrl+F -Wconversion to find an intersting place to fix

Open questions: on /performance/benchmarks

Regarding your
https://bloomberg.github.io/blazingmq/docs/performance/benchmarks
its a good report and thanks for publishing it and having following quesitons.

1. Did you run your scenaris with simulated network latency across 3 regions ( ASIA, Europe and America )
2. About Network card, Is it off the self NIC or any performance boosted network card ?

BlazingMQ

BlazingMQ a high-performance, reliable, and feature-rich message queueing system, has been released.

BlazingMQ provides a number of features that make it a powerful and versatile message queueing system, including:

Durability: Messages are persisted to disk, so they are not lost even if a broker or client fails.
Fault tolerance: BlazingMQ can tolerate the failure of individual brokers or clients without affecting the overall system.
High performance: BlazingMQ can handle millions of messages per second, making it suitable for high-volume applications.
High availability: BlazingMQ can be configured to be highly available, so that messages are always delivered even if some of the brokers are unavailable.
Rich feature set: BlazingMQ provides a number of features that are not available in other message queueing systems, such as support for various message routing strategies, compression, strong consistency, and poison pill detection.
BlazingMQ is a good choice for a variety of applications, including:

Real-time streaming applications
Distributed computing applications
Microservices applications
Event-driven applications
The release of BlazingMQ includes a number of new features and improvements, including:

Support for multiple message routing strategies
Compression support
Strong consistency support
Poison pill detection
Improved performance and scalability

Python SDK API Documentation showing spurious "page not found" errors.

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When performing the following search in the Blazing MQ documentation: https://bloomberg.github.io/blazingmq/docs/apidocs/python_apidocs/search.html?q=thread&check_keywords=yes&area=default

The synopsis for each match is a file not found error message. But the link to the document works. Below is sample text from the search results.

Search Results

Search finished, found 3 page(s) matching the search query.

Examples
...Page not foundThe page you requested could not be found. Try using the navigation or search to find what you're looking for or go to this site's home page.Back to topCopyright © 2023 Bloomberg

blazingmq
...Page not foundThe page you requested could not be found. Try using the navigation or search to find what you're looking for or go to this site's home page.Back to topCopyright © 2023 Bloomberg

Expected Behavior

To only display the error if there's an actual error.

Steps To Reproduce

This should be reproducible with the link above.

BlazingMQ Version

HEAD

Anything else?

This issue was reported in the Python SDK as bloomberg/blazingmq-sdk-python#11. The Python SDK documentation is hosted in this repository, so fixing it will involve a code change to the Python SDK codebase along with a PR here.

Ultimately it will make more sense to host the Python API docs on GH pages in the Python SDK repo.

Issue building the container on Mac OS with M2

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Having trouble building the image on - mac os venture Apple m2 chip

On running command - docker compose -f docker/single-node/docker-compose.yaml up --build -d

the script build_deps.sh in the container is failing to execute complaining

gcc: error: unrecognized command-line option ‘-m64’
ninja: build stopped: subcommand failed.

Attached screenshot for reference.

Expected Behavior

No response

Steps To Reproduce

running command -> docker compose -f docker/single-node/docker-compose.yaml up --build -d

BlazingMQ Version

1

Anything else?

Replace C-style type conversions with C++ casts

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

There are several places where C-style conversions are used.

Search the codebase for (double):

mwcst_printutil.cpp - 2
mwcst_statutil.cpp - 3
mwcu_printutil.t.cpp - 3

It's possible to find some conversions by searching for (int) too, but this search shows more false alarms and unrelated (int) usage. There are still conversions to be fixed, like:

int BasicTableInfoProvider::numColumns(int level) const
{
    if (level == 0) {
        return (int)d_columns.size();
    }
    else {
        return (int)d_columnGroups.size();
    }
}

Another possible searches: int(, double( (with a space in the beginning)

Describe the solution you'd like

Use static_cast

Warning: there might be some auto-generated files with C-style conversions, usually with a name ending with messages.cpp or messages.h, don't need to modify it.

Alternatives you considered

No response

Improve build documentation

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

Currently, we have two supported ways of building BlazingMQ:

1. Building using our build scripts in bin/ for Ubuntu and Darwin. These build scripts are very useful for managing some of the complexity of building our two biggest dependencies, bde and ntf-core, but for a user to build with them, they need to read through the build script first to ensure that they have the proper package manager dependencies installed. They also need to read through the script to see what environment variables they can pass to control the building and installation of BlazingMQ and its dependencies.
2. Building with our example Dockerfile, either in a single-node or cluster scenario. This also serves as a very useful example for deploying BlazingMQ, as it contains example configuration files.

Additionally, we may soon have vcpkg support coming. This will not make the above obsolete for all users, especially if they want to do development on the broker itself.

Other open source projects (for example, OBS, Kubernetes, or GNU/Emacs) maintain documentation that walks users through all supported ways of building the software. This is useful in addition to the above build scripts; Kubernetes, for instance, uses its build documentation to describe how to use the build scirpts.

Describe the solution you'd like

To lower the cost to entry for people wishing to contribute to the project, it would be beneficial to include better documentation on how to build and install a BlazingMQ broker and client library. While this is especially important with regard to the broker, it still is beneficial for users who would prefer building themselves, perhaps if they are packaging BlazingMQ or are in an environment where they aren't able to use our potential vcpkg support.

Alternatives you considered

No response

Cppcheck: remove statics in test `mqbsl_readwriteondisklog.t.cpp`

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

cppcheck output:

src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:945:9: error: Non-local variable 'g_bufferFactory_p' will use pointer to local variable 'bufferFactory'. [danglingLifetime]
        g_bufferFactory_p     = &bufferFactory;
        ^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:945:33: note: Address of variable taken here.
        g_bufferFactory_p     = &bufferFactory;
                                ^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:941:40: note: Variable created here.
        bdlbb::PooledBlobBufferFactory bufferFactory(k_LONG_ENTRY_LENGTH * 2,
                                       ^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:945:9: note: Non-local variable 'g_bufferFactory_p' will use pointer to local variable 'bufferFactory'.
        g_bufferFactory_p     = &bufferFactory;
        ^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:946:9: error: Non-local variable 'g_miniBufferFactory_p' will use pointer to local variable 'miniBufferFactory'. [danglingLifetime]
        g_miniBufferFactory_p = &miniBufferFactory;
        ^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:946:33: note: Address of variable taken here.
        g_miniBufferFactory_p = &miniBufferFactory;
                                ^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:943:40: note: Variable created here.
        bdlbb::PooledBlobBufferFactory miniBufferFactory(k_ENTRY_LENGTH,
                                       ^
src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp:946:9: note: Non-local variable 'g_miniBufferFactory_p' will use pointer to local variable 'miniBufferFactory'.
        g_miniBufferFactory_p = &miniBufferFactory;

Describe the solution you'd like

Will be nice to remove global objects construction from here:

blazingmq/src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp

Lines 941 to 946 in ca6491f

	bdlbb::PooledBlobBufferFactory bufferFactory(k_LONG_ENTRY_LENGTH * 2,
	s_allocator_p);
	bdlbb::PooledBlobBufferFactory miniBufferFactory(k_ENTRY_LENGTH,
	s_allocator_p);
	g_bufferFactory_p = &bufferFactory;
	g_miniBufferFactory_p = &miniBufferFactory;

And make it a field of the Tester class, which is constructed locally for each test.

blazingmq/src/groups/mqb/mqbsl/mqbsl_readwriteondisklog.t.cpp

Line 124 in ca6491f

struct Tester {

Current implementation has a good point - it builds these global objects once and saves some processing time on rebuilding it. But this also might be a problem, because it's a shared state among tests.

Alternatives you considered

No response

Session::nextEvent interprets timeouts under 1us as infinite

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Timeout under 1us (for example, 100ns) is interpreted as infinite timeout

Expected Behavior

Only zero timeout should be interpreted as "infinite"

Steps To Reproduce

Create a sync session
Pass 1ns as the timeout to Session::nextEvent
The call will block instead of returning immediately

BlazingMQ Version

1.13.3

Anything else?

No response

Evaluate Profile-Guided Optimization (PGO)

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

No - that's just an idea of how to improve the performance of BlazingMQ possibly.

Describe the solution you'd like

Recently I checked Profile-Guided Optimization (PGO) improvements on multiple projects. The results are here. I think these results are quite promising and can be a stimulus to test PGO with BlazingMQ.

We need to perform PGO benchmarks on BlazingMQ. And if it shows improvements (reduced CPU usage/reduced latency/anything else) - add a note to the documentation about possible improvements in BlazingMQ performance with PGO. Providing an easier way (e.g. a build option) to build scripts with PGO can be helpful for the end-users too.

Alternatives you considered

No response

Can't download bmqbrkr image from docker

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I'm attempting to start the single-node test project but when I run docker-compose I get the following:

docker compose -f docker/single-node/docker-compose.yaml up --build -d
[+] Running 0/1
 ⠿ bmqtool Error                                                                                                                                
Error response from daemon: pull access denied for bmqbrkr, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Expected Behavior

I expected the docker container(s) to start up.

Steps To Reproduce

> git clone [email protected]:bloomberg/blazingmq.git
> cd blazingmq
> docker login
> docker compose -f docker/single-node/docker-compose.yaml up --build -d

BlazingMQ Version

0.0.0

Anything else?

I checked out docker hub but I don't see the image bmqbrkr anywhere.

blazingmq how to increase scalability

I read the docs.

There seems blazingmq to be no support for adding machines to improve overall scalability like kafka(add more machines, increase partitions).

Possible to mis-use `QueueOptions` with subscriptions to cause broker crash

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

The sample consumer with the problem exposed:

https://github.com/bloomberg/blazingmq/pull/132/files

The crash log looks like this:

20OCT2023_13:24:39.771 (6108901376) FATAL mqbblp_queuehandle.cpp:634 Assertion failed: subscription->d_downstreamSubQueueId == downstreamSubId, stack trace:
(0): bmqAssertHandler(char const*, char const*, int)+0xe4 at 0x104461d84 in bmqbrkr.tsk
(1): BloombergLP::bsls::Assert::failOnViolation(BloombergLP::bsls::AssertViolation const&)+0x54 at 0x1053f82f4 in bmqbrkr.tsk
(2): BloombergLP::bsls::Assert::invokeHandler(BloombergLP::bsls::AssertViolation const&)+0x24 at 0x1053f857c in bmqbrkr.tsk
(3): BloombergLP::mqbblp::QueueHandle::registerSubscription(unsigned int, unsigned int, BloombergLP::bmqp_ctrlmsg::ConsumerInfo const&, unsigned int)+0x324 at 0x1048691a8 in bmqbrkr.tsk
(4): BloombergLP::mqbblp::Routers::Consumer::registerSubscriptions(BloombergLP::mqbi::QueueHandle*)+0xb8 at 0x10496acfc in bmqbrkr.tsk
(5): BloombergLP::mqbblp::Routers::AppContext::registerSubscriptions()+0x80 at 0x10496e640 in bmqbrkr.tsk
(6): BloombergLP::mqbblp::RootQueueEngine::configureHandle(BloombergLP::mqbi::QueueHandle*, BloombergLP::bmqp_ctrlmsg::StreamParameters const&, bsl::function<void (BloombergLP::bmqp_ctrlmsg::Status const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&)> const&)+0xe3c at 0x10494bd24 in bmqbrkr.tsk
(7): BloombergLP::mqbblp::LocalQueue::configureHandle(BloombergLP::mqbi::QueueHandle*, BloombergLP::bmqp_ctrlmsg::StreamParameters const&, bsl::function<void (BloombergLP::bmqp_ctrlmsg::Status const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&)> const&)+0xf8 at 0x1047e8e14 in bmqbrkr.tsk
(8): BloombergLP::mqbblp::Queue::configureHandle(BloombergLP::mqbi::QueueHandle*, BloombergLP::bmqp_ctrlmsg::StreamParameters const&, bsl::function<void (BloombergLP::bmqp_ctrlmsg::Status const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&)> const&)+0xb8 at 0x10480291c in bmqbrkr.tsk
(9): BloombergLP::mqbblp::QueueHandle::configureDispatched(BloombergLP::bmqp_ctrlmsg::StreamParameters const&, bsl::function<void (BloombergLP::bmqp_ctrlmsg::Status const&, BloombergLP::bmqp_ctrlmsg::StreamParameters const&)> const&)+0x344 at 0x10486bc00 in bmqbrkr.tsk

Expected Behavior

Will be good to handle this in the broker, so other SDKs (including custom-made) are not able to cause crash. Also will be good to add check for this in the SDK, so the troublesome message is not sent at all.

Steps To Reproduce

1. Build QueueOptions with subscriptions
2. Use the same QueueOptions for different fanout appIds.
3. Broker crashes.

BlazingMQ Version

0.90.18

Anything else?

No response

How to install the libs locally after done compiling it?

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

Successfully compiled the library, and now trying to compile the tutorial source code.

Describe the solution you'd like

I successfully compiled the library using the supplied sh script. Now how can I install the libraries in usual places (/usr/local/*) for local development?

Alternatives you considered

No response

Protocol specification.

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

I want to implement a client to your broker.

Describe the solution you'd like

Will there be a protocol specification added?

Alternatives you considered

No response

Add shell linting on GitHub actions

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

Currently our GitHub actions lack a linting step for shell scripts. Doing so might have been able to catch some of the comments of #34 and #54 before review.

Describe the solution you'd like

Proposed solution is to add a step similar to the clang-format step we already run on C++ code.

Alternatives you considered

No response

Compilation failed on Fedora 39.

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Compiling on Fedora 39 failed because of linking issue;

[100%] Linking CXX executable bmqtool.tsk
/usr/bin/ld: /usr/lib64/libfl.so: undefined reference to `yylex'

which could be traced to how flex 2.6.x changed the way yylex behaves.

Expected Behavior

No response

Steps To Reproduce

Env - OS: Fedora 39

Step - run bin/build-ubuntu.sh (commenting out the apt update and installations)

Expected - to compile/link without errors

Errors -

[100%] Linking CXX executable bmqtool.tsk
/usr/bin/ld: /usr/lib64/libfl.so: undefined reference to `yylex'

BlazingMQ Version

0.90.21

Anything else?

I solved the compilation issue by adding target_link_options(bmq PUBLIC "LINKER:-as-needed") to the CMakeLists.txt in the src/groups/bmq/ folder,

if(NOT CMAKE_HOST_SYSTEM_NAME MATCHES "Darwin")
  target_link_libraries(bmq PUBLIC "${FLEX_LIBRARIES}")
  target_link_options(bmq PUBLIC "LINKER:-as-needed")
endif()

However I am not sure if that's the right solution without affecting how BlazingMQ would behave.

Go and Python SDK

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

We build systems on cloud with C++, Java, Go and Python languages.

Describe the solution you'd like

Although Go and Python can interactive whith C++ SDK, but it would be nice if there is native support.

Alternatives you considered

No response

Help needed building blazingmq

Hi,

I am new here, trying to build the code on Ubuntu 20.04. I ran into an issue running cmake:

CMake Error at /usr/local/share/cmake/BdeBuildSystem/BdeEmitPkgConfigFile.cmake:17 (message):
You need to supply LIBDIR to bbs_emit_pc_files or set CMAKE_INSTALL_LIBDIR
Call Stack (most recent call first):
etc/cmake/TargetBMQStyleUor.cmake:119 (bbs_emit_pkgconfig_file)
src/groups/mqb/CMakeLists.txt:27 (bmq_emit_pkg_config)

I am not an expert user of CMake, can anybody tell me what I am missing?

Instructions for local Mac development of Just the Docs documentation is unclear

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

The docs/README.md file describes how to do local development of our Just the Docs documentation website on a Mac system, but has some stray references to tools that most users will not have access to.

Expected Behavior

Anyone should be able to follow the steps for local Mac development on our Just the Docs documentation, so they can easily submit patches.

Steps To Reproduce

.

BlazingMQ Version

HEAD

Anything else?

No response

Delayed message queue?!

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

Problem is for very long delayed queue.

Describe the solution you'd like

Delayed message queue that can hold millions of messages for long enough (e.g. 365 days), and persistent is required.

Alternatives you considered

With rabbitmq and delayed message queue plugin it's possible to achieve that partly, mostly because of limitations in mnesia.

Documentation is inconsistent on compression of message properties

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

In the Compression section of the documentation, the Summary section states:

Producer applications can indicate to the BlazingMQ SDK to compress a message. BlazingMQ SDK will compress the message (both payload and properties) before sending it to the BlazingMQ back-end.

Later on the same page in the Important Notes section, there is an apparent contradiction:

Only the message payload is compressed. Message properties are not compressed in order to ensure that the BlazingMQ message broker can read the properties efficiently if needed (for example, properties are read by the broker for evaluating subscriptions).

I'm assuming both of these can't be correct — properties cannot be both compressed and not compressed. (I mean they could be, but it would seem surprising)

Expected Behavior

The documentation is self-consistent.

Steps To Reproduce

1. Read the compression documentation.

BlazingMQ Version

1.0.0

Anything else?

No response

"Default" CMake Usage

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

The code is using internal bloomberg tools to find and build using the .mem and .dep definitions
that means that common tools that relies in cmake find_package to show and visualize cmake information will not be able to run with this project.

would it be ok to use the (documented) way to develop with bde, and add the find_package(bsl) int he places that are needed? This way we can package blazingmq together with the packaged version of bde-tools and bde.

Describe the solution you'd like

Use standard cmake for opensource builds

Alternatives you considered

No response

Fix ShellCheck failures

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

The new ShellCheck linter job fails on three existing shell scripts in our repository:

• ./bin/build-ubuntu.sh,
• ./bin/build-darwin.sh, and
• ./src/applications/bmqbrkr/run.

Expected Behavior

CI checks should not fail.

Steps To Reproduce

https://github.com/bloomberg/blazingmq/actions/runs/5755158855/job/15602005338

BlazingMQ Version

HEAD

Anything else?

No response

build_bde failing

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

On running:
$ docker compose -f docker/single-node/docker-compose.yaml up --build -d

Getting the following error:

5.847 + bbs_build configure
5.847 /workspace/srcs/bde /workspace
5.873 qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory
5.873 Error: Command '['cmake', '/workspace/srcs/bde', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=Release', '-DBDE_BUILD_TARGET_64=1', '-DBDE_BUILD_TARGET_CPP17=1', '-DBdeBuildSystem_DIR:PATH=/workspace/srcs/bde-tools/BdeBuildSystem', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DBBS_BUILD_SYSTEM=ON', '-DBBS_USE_WAFSTYLEOUT=OFF', '-DBBS_CPP11_VERIFY_NO_CHANGE=OFF', '-DCMAKE_INSTALL_PREFIX=/opt/bb', '-DCMAKE_INSTALL_LIBDIR=lib64', '-DCMAKE_TOOLCHAIN_FILE=/workspace/srcs/bde-tools/BdeBuildSystem/toolchains/linux/gcc-default.cmake', '-DCMAKE_PREFIX_PATH:PATH=///opt/bb', '--log-level=ERROR']' returned non-zero exit status 255.
5.873 Configuration cmd:
5.873 cmake /workspace/srcs/bde -G Ninja -DCMAKE_BUILD_TYPE=Release -DBDE_BUILD_TARGET_64=1 -DBDE_BUILD_TARGET_CPP17=1 -DBdeBuildSystem_DIR:PATH=/workspace/srcs/bde-tools/BdeBuildSystem -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBBS_BUILD_SYSTEM=ON -DBBS_USE_WAFSTYLEOUT=OFF -DBBS_CPP11_VERIFY_NO_CHANGE=OFF -DCMAKE_INSTALL_PREFIX=/opt/bb -DCMAKE_INSTALL_LIBDIR=lib64 -DCMAKE_TOOLCHAIN_FILE=/workspace/srcs/bde-tools/BdeBuildSystem/toolchains/linux/gcc-default.cmake -DCMAKE_PREFIX_PATH:PATH=///opt/bb --log-level=ERROR
------
failed to solve: process "/bin/sh -c bin/build_deps.sh" did not complete successfully: exit code: 1

Laptop spec:
Docker Version: 24.0.6
Macbook M2 Air

Expected Behavior

Build to complete without any issues

Steps To Reproduce

1. Clone the repository
2. Follow the first step here: https://bloomberg.github.io/blazingmq/docs/getting_started/blazingmq_in_action/
   3

BlazingMQ Version

1.13.3

Anything else?

N/A

PURGE QUEUE / PURGE DOMAIN commands do not purge inactive queues

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Currently, we have 2 different states for an existing queue: active and inactive.

• Active queue is a queue which had a recent client connection to the BlazingMQ cluster.
• Inactive queue is a queue which has some records stored in a journal, but it doesn't have recent clients.

The difference between these 2 queues is that active queues have both storage level representation and runtime object representation mqbi::Queue, while inactive queue has only storage level representation.

The current implementation of PURGE QUEUE / PURGE DOMAIN iterates through mqbi::Queue objects so it is not aware of inactive queues at all. The implementation is not able to purge inactive queues.

Expected Behavior

The implementation should work on a storage level instead, where we have all the information needed to purge both active and inactive queues.

Steps To Reproduce

1. Start the BlazingMQ cluster.
2. Start the producer and PUT several messages to a queue.
3. Stop the producer and the BlazingMQ cluster.
4. Start the BlazingMQ cluster again.
5. Send ADMIN COMMAND to purge the queue with messages or the corresponding domain - posted messages will never be purged, because the queue is currently inactive.

BlazingMQ Version

0.90.20

Anything else?

No response

Fix: add checks to pointer arguments in `m_bmqtool_inpututil`

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

Files: m_bmqtool_inpututil.h and m_bmqtool_inpututil.cpp

There are functions with arguments passed as a pointer, with assumption that these pointer values are not null.

Example:

blazingmq/src/applications/bmqtool/m_bmqtool_inpututil.cpp

Lines 50 to 53 in 9b692fe

	void InputUtil::preprocessInput(bsl::string* verb,
	bsl::string* output,
	const bsl::string& input,
	bsl::unordered_set<bsl::string>* keys)

Describe the solution you'd like

Add checks to pointer arguments to functions in m_bmqtool_inpututil.

Example:

blazingmq/src/groups/mqb/mqbmock/mqbmock_cluster.cpp

Lines 380 to 384 in 9b692fe

	Cluster::sendRequest(const Cluster::RequestManagerType::RequestSp& request,
	mqbnet::ClusterNode* target,
	bsls::TimeInterval timeout)
	{
	BSLS_ASSERT_SAFE(target);

Alternatives you considered

No response

Subscriptions expressions: More powerful pattern matching on string-typed properties

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

There is currently a single useful operator for string properties (== and !=). For use-cases that need more powerful pattern matching, the following operations would be useful:

• Checking that a string contains another string
• Checking that a string starts with another string or a character
• Doing pattern matching on a sub-section of an URI (example: /var/log/foobar/*/*.log.)

Describe the solution you'd like

A regex operator would solve many of those problems. We can use the following library to compile regexes without introducing additional dependencies to the broker:
https://github.com/bloomberg/bde/blob/d6674e1df52b62d66942ca318882f225d26104bf/groups/bdl/bdlpcre/bdlpcre_regex.h#L669

Alternatives you considered

No response

Docker compose doesn't work on Apple M2 MacBook Pro

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When running the initial docker compose command on an M2 silicon MacBook Pro, the build fails with the following error:

36.19 rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2

Expected Behavior

Docker compose completes successfully with no errors.

Steps To Reproduce

1. Check out repository on M2 silicon MacBook Pro
2. Enter root of repository
3. Run docker compose -f docker/single-node/docker-compose.yaml up --build -d

BlazingMQ Version

4be4902

Anything else?

Here is the error in context:

FROM --platform=linux/amd64 docker.io/ubuntu:22.04 AS builder
36.16 ++ BDE_CMAKE_UFID=opt_64_cpp17
36.16 ++ export BDE_CMAKE_BUILD_DIR=_build/unix-linux-aarch64-6.4.16-gcc-11.4.0-opt_64_cpp17
36.16 ++ BDE_CMAKE_BUILD_DIR=_build/unix-linux-aarch64-6.4.16-gcc-11.4.0-opt_64_cpp17
36.16 ++ export CC=/usr/bin/gcc
36.16 ++ CC=/usr/bin/gcc
36.16 ++ export CXX=/usr/bin/g++
36.16 ++ CXX=/usr/bin/g++
36.16 ++ export BDE_CMAKE_TOOLCHAIN=/workspace/srcs/bde-tools/BdeBuildSystem/toolchains/linux/gcc-default.cmake
36.16 ++ BDE_CMAKE_TOOLCHAIN=/workspace/srcs/bde-tools/BdeBuildSystem/toolchains/linux/gcc-default.cmake
36.16 ++ export BDE_CMAKE_INSTALL_DIR=/workspace/_install
36.16 ++ BDE_CMAKE_INSTALL_DIR=/workspace/_install
36.16 + build
36.16 + build_bde
36.16 + pushd srcs/bde
36.16 /workspace/srcs/bde /workspace
36.16 + bbs_build configure
36.19 rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2
36.19  Error: Command '['cmake', '/workspace/srcs/bde', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=Release', '-DBDE_BUILD_TARGET_64=1', '-DBDE_BUILD_TARGET_CPP17=1', '-DBdeBuildSystem_DIR:PATH=/workspace/srcs/bde-tools/BdeBuildSystem', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DBBS_BUILD_SYSTEM=ON', '-DBBS_USE_WAFSTYLEOUT=OFF', '-DBBS_CPP11_VERIFY_NO_CHANGE=OFF', '-DCMAKE_INSTALL_PREFIX=/opt/bb', '-DCMAKE_INSTALL_LIBDIR=lib64', '-DCMAKE_TOOLCHAIN_FILE=/workspace/srcs/bde-tools/BdeBuildSystem/toolchains/linux/gcc-default.cmake', '-DCMAKE_PREFIX_PATH:PATH=///opt/bb', '--log-level=ERROR']' died with <Signals.SIGTRAP: 5>.
36.19 Configuration cmd:
36.19 cmake /workspace/srcs/bde -G Ninja -DCMAKE_BUILD_TYPE=Release -DBDE_BUILD_TARGET_64=1 -DBDE_BUILD_TARGET_CPP17=1 -DBdeBuildSystem_DIR:PATH=/workspace/srcs/bde-tools/BdeBuildSystem -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBBS_BUILD_SYSTEM=ON -DBBS_USE_WAFSTYLEOUT=OFF -DBBS_CPP11_VERIFY_NO_CHANGE=OFF -DCMAKE_INSTALL_PREFIX=/opt/bb -DCMAKE_INSTALL_LIBDIR=lib64 -DCMAKE_TOOLCHAIN_FILE=/workspace/srcs/bde-tools/BdeBuildSystem/toolchains/linux/gcc-default.cmake -DCMAKE_PREFIX_PATH:PATH=///opt/bb --log-level=ERROR

Remove unused header mwcsys_executil.h from the codebase

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

If you search for ExecUtil usage, you can find some files with unused #include <mwcsys_executil.h>:
https://github.com/search?q=repo%3Abloomberg%2Fblazingmq%20ExecUtil&type=code

Describe the solution you'd like

Will be good to remove these unused includes where possible, while keeping files mwcsys_executil.h, mwcsys_executil.cpp, mwcsys_executil.t.cpp. These files might be used by existing library's clients.

Alternatives you considered

No response

Refactor: make AdminClient auto-closeable AbstractContextManager

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

Currently, AdminClient is not suitable to use in with statements ensuring auto freeing of resources.

blazingmq/src/python/blazingmq/dev/it/process/admin.py

Line 16 in 6f1321d

class AdminClient:

Describe the solution you'd like

Implement AbstractContextManager interfaces in AdminClient

Alternatives you considered

No response

Change `k_DEFAULT_SUBSCRIPTION_ID` to `k_DEFAULT_SUBQUEUE_ID`

blazingmq/src/groups/mqb/mqbblp/mqbblp_relayqueueengine.cpp

Line 1648 in 7192789

*subQueueId = bmqp::Protocol::k_DEFAULT_SUBSCRIPTION_ID;

Both of these constants happen to have the value 0, but in principle we should be assigning the default subqueue ID to a subqueue id variable, rather than the default subscription id to a subqueue id variable.

GitHub Actions CI warns of deprecated Node.js 16 actions

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Our tests are currently using older versions of certain GitHub Actions which rely on the outdated Node.js version 16:

Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/checkout@v3, actions/cache@v3. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.

This does not cause the tests to fail but adds noisy warnings to Actions output.

Expected Behavior

No response

Steps To Reproduce

This can be seen on any run of the tests workflow, as in: https://github.com/bloomberg/blazingmq/actions/runs/7805114652

BlazingMQ Version

HEAD

Anything else?

These warnings can be fixed by updating to actions/checkout@v4 and actions/cache@v4, which do not have any breaking changes over our current versions. A similar change has already been done in the Python SDK: bloomberg/blazingmq-sdk-python@4944dcf

Proposal: simplify throttled logging in the codebase

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

Currently we have 3 ways to throttle logging:

1. With mwcu::ThrottledActionParams:

   blazingmq/src/groups/mqb/mqbblp/mqbblp_cluster.cpp

   Lines 414 to 419 in 5f65a2a

   	// Throttle error log if this is a 'failed Ack': note that we log at
   	// INFO level in order not to overwhelm the dashboard, if a queue is
   	// full, every post will nack, which could be a lot.
   	MWCU_THROTTLEDACTION_THROTTLE(
   	d_throttledFailedAckMessages,
   	BALL_LOG_INFO << description() << ": failed Ack "

2. With bdlmt::Throttle:
   

   blazingmq/src/groups/mqb/mqbs/mqbs_filestore.cpp

   Line 3669 in 5f65a2a

   if (d_alarmSoftLimiter.requestPermission()) {

3. With BALL_LOGTHROTTLE:

   blazingmq/src/groups/mqb/mqbblp/mqbblp_relayqueueengine.cpp

   Line 330 in 5f65a2a

   BALL_LOGTHROTTLE_INFO_BLOCK(k_MAX_INSTANT_MESSAGES, k_NS_PER_MESSAGE)

The good example of a problem: mqbblp::Cluster, here we have many-many fields responsible for throttled logging in the header:

blazingmq/src/groups/mqb/mqbblp/mqbblp_cluster.h

Lines 254 to 292 in 5f65a2a

	mwcu::ThrottledActionParams d_throttledFailedPutMessages;
	// Throttling parameters for failed PUT
	// messages.
	mwcu::ThrottledActionParams d_throttledSkippedPutMessages;
	// Throttling parameters for dropped
	// PUT messages.
	mwcu::ThrottledActionParams d_throttledFailedAckMessages;
	// Throttling parameters for failed ACK
	// messages.
	mwcu::ThrottledActionParams d_throttledDroppedAckMessages;
	// Throttling parameters for dropped
	// ACK messages.
	mwcu::ThrottledActionParams d_throttledFailedConfirmMessages;
	// Throttling parameters for failed
	// CONFIRM messages.
	mwcu::ThrottledActionParams d_throttledFailedRejectMessages;
	// Throttling parameters for failed
	// REJECT messages.
	mwcu::ThrottledActionParams d_throttledDroppedConfirmMessages;
	// Throttling parameters for dropped
	// CONFIRM messages.
	mwcu::ThrottledActionParams d_throttledDroppedRejectMessages;
	// Throttling parameters for dropped
	// REJECT messages.
	mwcu::ThrottledActionParams d_throttledFailedPushMessages;
	// Throttling parameters for failed
	// PUSH messages.
	mwcu::ThrottledActionParams d_throttledDroppedPushMessages;
	// Throttling parameters for dropped
	// PUSH messages.

Removing these fields by using a local BALL_LOGTHROTTLE will greatly simplify the class.

Describe the solution you'd like

Replace field-stored mwcu::ThrottledActionParams with local BALL_LOGTHROTTLE where it's possible.

Alternatives you considered

No response

HTTP webhook integration

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

The ability to send JSON via HTTP posts to a wehbook endpoint which then takes that JSON and adds it to the message queue.

Describe the solution you'd like

I know MQ's + HTTP is kind of an antipatern and usually the advice is to write your own middleware. So far I have not seen a message queue that offers this feature, but it seems like a common enough problem that people should not have to write their own customer middleware every time.

An example of this is can be found trying to search for ways to integrate HTTP messages with RabbitMQ and finding this project https://github.com/dougwatson/http2amqp

The ideal scenario would be if you could create a webhook endpoint on BlazingMQ which would then be able to receive JSON messages from HTTP (push/pub) sources which would convert the JSON into a message and place it into a queue.

Alternatives you considered

Custom middleware

Possible `int` overflow in `bmqt::UriParser`

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

No response

Expected Behavior

No response

Steps To Reproduce

However, I can see one very unprobable scenario with a possible bug here.

1. Within UriParse::initialize we increment int s_initialized counter in a thread-safely manner (within the locked mutex).
   

   blazingmq/src/groups/bmq/bmqt/bmqt_uri.cpp

   Line 195 in 342b608

   ++s_initialized;

2. We early return if this counter is greater than 1, meaning, that some other code has already incremented it and initialized a shared state which is const-read-thread-safe.

3. However, we do bmqt::UriParse::initialize every time we call a constructor for bmqt::Uri, and do not deinitialize bmqt::UriParser until we call a destructor bmqt::Uri::~Uri.

4. This means, that if we are able to keep approx 2^31 bmqt::Uris in memory, we'll get int overflow.

5a. Because of this possible overflow, the condition for early exit will fail on a new bmqt::Uri construction, and we might re-initialize some const-read-thread-safe RegEx, when some other thread might use it

5b. It is also possible to just launch bmqt::UriParser::initialize 2.2kkk times to get overflow. Even if we do call bmqt::UriParser::shutdown the same number of times afterwards (according to contract), there will be a problem.

BlazingMQ Version

0.90.21

Anything else?

There are 2 possible solutions.

1. The minimal change (and safe) solution is to change the counter type int -> bsls::Types::Int64, so it will be impossible to reach overflow.

2. The good solution is to give up on a custom reference counting of bmqt::UriParser::initialize and initialize it once and for all as a singleton, using the appropriate tool to initialize: BSLMT_ONCE_DO https://bloomberg.github.io/bde-resources/doxygen/bde_api_prod/group__bslmt__once.html

The greatest benefit of this approach is that we don't lock the mutex every time we need to build a bmqt::Uri. As you can see here (https://github.com/bloomberg/bde/blob/88f20d8057d462e2bdaf27b757ff3bf335555dcd/groups/bsl/bslmt/bslmt_once.cpp#L25), bslmt::Once can check the need to do something without locking the internal mutex, when the state of once was already decided. As a result, we'll be able to work with bmqt::Uris much, much faster from different threads.

The difference: we'll not free these resources gracefully to make the allocator happy in the end of the program. I don't think this difference is bad, because the needed allocated memory is small, we already have some singletons allocated like this, and the runtime performance gain is huge.

We can also deprecate bmqt::UriParser::initialize, bmqt::UriParser::shutdown or make them no-op.

Fix static local variable multi-threaded initialization

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Place where the problem might occur:

blazingmq/src/groups/bmq/bmqt/bmqt_subscription.cpp

Line 46 in a6f119b

static bsls::AtomicUint s_id = 0;

This static local variable initialization is not thread safe in general. The probability to observe a bug is small though.

Here are some details and advices:

https://bloomberg.github.io/bde-resources/doxygen/bde_api_prod/group__bslmt__once.html#_details

Expected Behavior

Here is how we did a safe initialization for another component:

blazingmq/src/groups/bmq/bmqt/bmqt_correlationid.cpp

Lines 33 to 42 in 08a6d16

	CorrelationId CorrelationId::autoValue()
	{
	static bsls::AtomicInt* g_id_p = 0; // A unique id for each AutoValue
	BSLMT_ONCE_DO
	{
	g_id_p = new bsls::AtomicInt(0);
	// Heap allocate it to prevent 'exit-time-destructor needed' compiler
	// warning. Causes valgrind-reported memory leak.
	}

Will be good to do the same for the problematic line of code above.

Steps To Reproduce

Try to build multiple SubscriptionHandles at the same time from different threads just after the app starts.

BlazingMQ Version

0.90.18

Anything else?

No response

NodeJS sdk

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

No

Describe the solution you'd like

I would like to have a library for using blazingMQ with NodeJS

Alternatives you considered

No response

Allow API documentation to be built directly from Doxygen

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Our current API documentation was generated using bde-tools's bdedox executable. Since our comments are no longer in BDE-style, running doxygen on the codebase does not pick up several things our current published API docs have, like package group, package, and component-level documentation, some BDE formatting, and \brief descriptions automatically generated from the first sentence of a documentation comment.

Additionally, some BDE-style documentation failed to convert, as in the bmqa_abstractsession component.

Expected Behavior

Running doxygen on the codebase should produce up-to-date documentation, including all our package group, package, and component documentation, and should support Doxygen-style annotations, such as \deprecated and \par.

Steps To Reproduce

1. Run doxygen on the codebase.
2. Compare the output to the public API documentation.

BlazingMQ Version

HEAD

Anything else?

No response

why the codebase so large?

No third-party libraries included，blazingmq codebase use cloc output.

dotnet SDK

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

No

Describe the solution you'd like

According to the documentation, clients for Java and Python are available, such as the following:

https://github.com/bloomberg/blazingmq-sdk-java

I would be interested in helping contribute to an sdk for dotnet if the project maintainers feel this might be of use to the broader community.

Alternatives you considered

No response

CI: add GitHub Actions check for Apple Silicon (M1) build

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

We don't have automatic checks for M1/M2 builds.

Describe the solution you'd like

According to this roadmap, Apple Silicon (M1) is in public beta in GitHub Actions:

github/roadmap#528

Might be possible to use the corresponding runner to check darwin build script automatically.

Alternatives you considered

No response

Screen frozen when running docker compose

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

docker compose is freezing see below when running. the machine i'm using is a ec2 machine: t2.micro

ubuntu@ip-172-31-85-234:~/development/blazingmq$ docker compose -f docker/single-node/do                                                                                                                      cker-compose.yaml up --build -d
[+] Running 1/1
 ✘ bmqtool Error                                                                   0.1s
[+] Building 92.6s (10/20)
 => [bmqbrkr internal] load build definition from Dockerfile                                                                                                                                             0.0s
 => => transferring dockerfile: 2.13kB                                                                                                                                                                   0.0s
 => [bmqbrkr internal] load .dockerignore                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                          0.0s
 => [bmqbrkr internal] load metadata for docker.io/library/ubuntu:22.04                                                                                                                                  0.2s
 => CACHED [bmqbrkr builder  1/12] FROM docker.io/library/ubuntu:22.04@sha256:0bced47fffa3361afa981854fcabcd4577cd43cebbb808cea2b1f33a3dd7f508                                                           0.0s
 => [bmqbrkr internal] load build context                                                                                                                                                                0.2s
 => => transferring context: 74.84kB                                                                                                                                                                     0.2s
 => CACHED [bmqbrkr builder  2/12] RUN apt update &&     apt -y install ca-certificates                                                                                                                  0.0s
 => CACHED [bmqbrkr builder  3/12] RUN apt install -y --no-install-recommends     build-essential     gdb     curl     python3.10     ninja-build     pkg-config     bison     libfl-dev     libbenchma  0.0s
 => CACHED [bmqbrkr builder  4/12] RUN mkdir -p /deps_build/CMake     && cd /deps_build/CMake     && curl -s -L -O https://github.com/Kitware/CMake/releases/download/v3.27.1/cmake-3.27.1-linux-x86_64  0.0s
 => CACHED [bmqbrkr builder  5/12] WORKDIR /workspace                                                                                                                                                    0.0s
 => CACHED [bmqbrkr builder  6/12] COPY docker/build_deps.sh bin/build_deps.sh                                                                                                                           0.0s
 => [bmqbrkr builder  7/12] RUN bin/build_deps.sh                                                                                                                                                       92.1s
 => => # [487/1488] Generating /workspace/srcs/bde/groups/bsl/bslma/bslma_sharedptrinplacerep_cpp03.h
 => => # [488/1488] Generating /workspace/srcs/bde/groups/bsl/bslma/bslma_stdallocator_cpp03.h
 => => # [489/1488] Generating /workspace/srcs/bde/groups/bsl/bslma/bslma_allocatortraits_cpp03.h
 => => # [490/1488] Generating /workspace/srcs/bde/groups/bsl/bslma/bslma_managedptr_cpp03.h
 => => # [491/1488] Generating /workspace/srcs/bde/groups/bsl/bslma/bslma_constructionutil_cpp03.h
 => => # [492/1488] Building CXX object groups/bsl/CMakeFiles/bslh-iface.dir/bslh/bslh_fibonaccibadhashwrapper.cpp.o

Expected Behavior

container should be able to be created

Steps To Reproduce

1. create a ec2: t2.micro machine
2. ssh from a client like mobaexterm or visual studio code
3. clone the code
4. perform docker compose -f docker/single-node/docker-compose.yaml up --build -d

BlazingMQ Version

1.0

Anything else?

No response

`bmqa_mocksession`'s `expect_XYZ` functions have different `const` qualifiers from their corresponding `XYZ` functions in `bmqa_abstractsession`

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

The expect_XYZ functions in bmqa_mocksession (e.g., expect_configureQueueSync) are run using the BMQA_EXPECT_CALL macro from a mock, and are meant to parallel the functions XYZ (e.g., configureQueueSync) in bmqa_abstractsession. At the moment, several of these functions expect_XYZ differ from their corresponding XYZ functions in the constness of their parameters:

• expect_closeQueueSync(QueueId*, ...), but closeQueueSync(const QueueId*, ...)
• expect_closeQueueAsync(QueueId*, ...), but closeQueueAsync(const QueueId*, ...)
• expect_configureQueueSync(QueueId*, ...), but configureQueueSync(const QueueId*, ...)

As a result, it is not possible to use BMQA_EXPECT_CALL from a mock session that implements the AbstractSession interface.

Expected Behavior

The arguments to expect_XYZ functions should match the arguments to XYZ functions in their constness.

Steps To Reproduce

.

BlazingMQ Version

HEAD

Anything else?

No response

C Client API (Wrapper over the C++ API)

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

C API is very useful in general, and can become a bridge for API in other languages as well.

Describe the solution you'd like

Since implementing a C API from ground-up would be too much effort, we plan to write a C wrapper over the existing C++ API.

Alternatives you considered

No response

Remove message group ID support

This feature is not supported in non-developer releases, and its use cases have been mostly superseded by BlazingMQ subscriptions.

Advantages over zmq ?

Hi there, first time hearing about this project,
I am used to zmq and zeroless to either send heavy data msgs or bulk async request, and I have easy python wrappers to interface with

Can you please explain how is is blazingmq better, in terms of benchmarks performance, features, and other metrics ?

Excited to experiment with it
Thanks a lot and all the best!

Misleading text on the Website

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

The website says that Blazingmq was developed entirely from the ground up without the use of any external library.
but that's not true, as soon as you open:
blazingmq/src/groups/mwc/grou[/mwc.dep:

you see that it's using BDE:

bal bdl bsl

Expected Behavior

No response

Steps To Reproduce

2nd paragraph of the website states that there's not a single external library. But it uses BDE.

BlazingMQ Version

1.0.0

Anything else?

No response

Issue with build script

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Build script(both build_ubuntu.sh and build_darwin.sh) fail to build blazingmq. Build produces a warning and fails to continue build due to flag set to fail on warnings. The following error occurs:
In file included from /home/jon/cs/bb/simon-fork/blazingmq/include/bslstl_ownerless.h:72, from /home/jon/cs/bb/simon-fork/blazingmq/include/bsl_memory.h:152, from /home/jon/cs/bb/simon-fork/blazingmq/include/bdlf_bind.h:900, from /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntccfg/ntccfg_bind.h:40, from /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntccfg/ntccfg_platform.h:22, from /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.h:22, from /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp:16: In destructor ‘bsl::shared_ptr<ELEMENT_TYPE>::~shared_ptr() [with ELEMENT_TYPE = BloombergLP::ntci::Executor]’, inlined from ‘BloombergLP::ntcs::ObserverRef<TYPE>::~ObserverRef() [with TYPE = BloombergLP::ntci::Executor]’ at /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_observer.h:533:32, inlined from ‘virtual void BloombergLP::ntcs::Strand::moveAndExecute(BloombergLP::ntci::Executor::FunctorSequence*, const BloombergLP::ntci::Executor::Functor&)’ at /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp:263:5: /home/jon/cs/bb/simon-fork/blazingmq/include/bslstl_sharedptr.h:5045:28: error: ‘*(bsl::shared_ptr<BloombergLP::ntci::Executor>*)((char*)&executorRef + offsetof(BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>,BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>::d_shared)).bsl::shared_ptr<BloombergLP::ntci::Executor>::d_rep_p’ may be used uninitialized [-Werror=maybe-uninitialized] 5045 | d_rep_p->releaseRef(); | ~~~~~~~~~~~~~~~~~~~^~ /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp: In member function ‘virtual void BloombergLP::ntcs::Strand::moveAndExecute(BloombergLP::ntci::Executor::FunctorSequence*, const BloombergLP::ntci::Executor::Functor&)’: /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp:254:43: note: ‘*(bsl::shared_ptr<BloombergLP::ntci::Executor>*)((char*)&executorRef + offsetof(BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>,BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>::d_shared)).bsl::shared_ptr<BloombergLP::ntci::Executor>::d_rep_p’ was declared here 254 | ntcs::ObserverRef<ntci::Executor> executorRef(&d_executor); | ^~~~~~~~~~~ In destructor ‘bsl::shared_ptr<ELEMENT_TYPE>::~shared_ptr() [with ELEMENT_TYPE = BloombergLP::ntci::Executor]’, inlined from ‘BloombergLP::ntcs::ObserverRef<TYPE>::~ObserverRef() [with TYPE = BloombergLP::ntci::Executor]’ at /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_observer.h:533:32, inlined from ‘virtual void BloombergLP::ntcs::Strand::execute(const BloombergLP::ntci::Executor::Functor&)’ at /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp:230:5: /home/jon/cs/bb/simon-fork/blazingmq/include/bslstl_sharedptr.h:5045:28: error: ‘*(bsl::shared_ptr<BloombergLP::ntci::Executor>*)((char*)&executorRef + offsetof(BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>,BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>::d_shared)).bsl::shared_ptr<BloombergLP::ntci::Executor>::d_rep_p’ may be used uninitialized [-Werror=maybe-uninitialized] 5045 | d_rep_p->releaseRef(); | ~~~~~~~~~~~~~~~~~~~^~ /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp: In member function ‘virtual void BloombergLP::ntcs::Strand::execute(const BloombergLP::ntci::Executor::Functor&)’: /home/jon/cs/bb/simon-fork/blazingmq/thirdparty/ntf-core/groups/ntc/ntcs/ntcs_strand.cpp:221:43: note: ‘*(bsl::shared_ptr<BloombergLP::ntci::Executor>*)((char*)&executorRef + offsetof(BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>,BloombergLP::ntcs::ObserverRef<BloombergLP::ntci::Executor>::d_shared)).bsl::shared_ptr<BloombergLP::ntci::Executor>::d_rep_p’ was declared here 221 | ntcs::ObserverRef<ntci::Executor> executorRef(&d_executor); | ^~~~~~~~~~~ cc1plus: all warnings being treated as errors gmake[3]: *** [CMakeFiles/ntcs.dir/build.make:594: CMakeFiles/ntcs.dir/groups/ntc/ntcs/ntcs_strand.cpp.o] Error 1 gmake[3]: *** Waiting for unfinished jobs.... gmake[2]: *** [CMakeFiles/Makefile2:7198: CMakeFiles/ntcs.dir/all] Error 2 gmake[1]: *** [Makefile:166: all] Error 2

This results in the following failure:

CMake Error at /home/jon/cs/bb/simon-fork/blazingmq/build/ntf/Linux/opt_dbg_exc_mt_64/cmake_install.cmake:1384 (file): file INSTALL cannot find "/home/jon/cs/bb/simon-fork/blazingmq/build/ntf/Linux/opt_dbg_exc_mt_64/lib/libntc.a": No such file or directory.

Expected Behavior

No response

Steps To Reproduce

1. Clone Blazingmq
2. Run ./bin/build-darwin.sh or ./bin/build-ubuntu.sh

BlazingMQ Version

1.0.0

Anything else?

Issue was encountered on a intel macbook as well as a machine running the Fedora linux distro

Subscriptions expressions: Ability to check whether a property exists

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

The expression language currently does not provide a clear way to check for the existence of a property.

Let's assume I have a queue with two types of messages.

Messages of type A have the following properties:

radius = 34
color = "blue"

Messages of type B have the following properties:

height: 53
width: 100
color = "red"

If I want to filter for messages that contain a radius field in a subscription, I can currently recur to the following workaround expressions:

radius == radius
radius >= 0 || radius < 0

however my intent to check for existence is not clear.

Describe the solution you'd like

I would love if the expression language allowed me to check for existence of fields explicitly. This can be done by either checking if a field:

!= NULL

for the existence check.

and

== NULL

for the missing property check.

Alternatively an operator for exists() can be introduced. This alternative would require a unary negation operator.

Alternatives you considered

No response