why the codebase so large?

[billowqiu]

why the codebase so large?

[pniedzielski]

Hi @billowqiu,

Thank you for your interest in BlazingMQ! BlazingMQ is feature-rich, battle-tested message queue that supports several different network topologies and message routing strategies, among much other functionality that we have found very important to our use cases. With this level of functionality necessarily comes a larger code footprint.

Some of what you're seeing is generated files for our protocol; when we exclude those, the size shrinks significantly:

src $ cloc --not-match-f="(_messages\\.|\\.t\\.cpp)" .
     865 text files.
     788 unique files.
      77 files ignored.

github.com/AlDanial/cloc v 1.98  T=0.55 s (1434.4 files/s, 609002.7 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
C++                             360          25858          25877         114699
C/C++ Header                    353          26594          63323          70271
XSD                               6            407            124           3908
Text                             36            248              0           1383
Markdown                         10             74              0            539
JSON                              9              0              0            465
CMake                            11             73            142            224
yacc                              1             22             15            127
lex                               1             32              4            118
Bourne Again Shell                1              8              5             29
--------------------------------------------------------------------------------
SUM:                            788          53316          89490         191763
--------------------------------------------------------------------------------

Another part of this is, as you say, because BlazingMQ is a C++03 codebase that depends on very few third-party libraries, we have needed to implement some of the basic, non-message-queue functionality that projects written in other languages might be able to pull in as a dependency more easily than we were able when BlazingMQ was first being written years ago. These components are contained within src/groups/mwc/, and represent a bit over one fifth of the codebase:

src $ cloc --not-match-f="(_messages\\.|\\.t\\.cpp)" groups/mwc/
     267 text files.
     242 unique files.
      25 files ignored.

github.com/AlDanial/cloc v 1.98  T=0.16 s (1507.9 files/s, 471296.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C/C++ Header                   112           8082          20127          25810
C++                            112           3318           3656          13592
Text                            13             76              0            507
Markdown                         3             28              0            236
XSD                              1             11              1            161
CMake                            1              5              9             18
-------------------------------------------------------------------------------
SUM:                           242          11520          23793          40324
-------------------------------------------------------------------------------

This repository also includes both the client C++ SDK libbmq, the broker bmqbrkr, and tools like bmqtool to interact with the broker outside of an application.

We don't believe this is too much to be worried about. Our peer open-source message queue solutions with comparable levels of functionality seem to be in the same order of magnitude of codebase size as us. A very quick check for Kafka:

kafka $ cloc *
    5642 text files.
    5501 unique files.
     198 files ignored.

github.com/AlDanial/cloc v 1.98  T=4.57 s (1203.6 files/s, 268274.4 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Java                           4282         116548         166894         632634
Scala                           633          30973          32410         173700
HTML                             60           2495           1158          21282
Python                          134           3809           5332          14968
JSON                            209            216              0           9182
Gradle                            4            507            308           2756
XML                              11            300            350           1680
Bourne Shell                     56            293           1006           1274
Markdown                         12            409              0           1129
Properties                       56            329           1263            709
Bourne Again Shell                2             66            112            497
DOS Batch                        35             79            520            266
Maven                             3             27             50            221
XSLT                              1             26             27            153
Dockerfile                        1             16             32             72
JavaScript                        1              3             15              6
Text                              1              0              0              1
--------------------------------------------------------------------------------
SUM:                           5501         156096         209477         860530
--------------------------------------------------------------------------------

And RabbitMQ (written in a less verbose language than us):

rabbitmq-server $ cloc *
    3104 text files.
    2540 unique files.
     636 files ignored.

github.com/AlDanial/cloc v 1.98  T=1.23 s (2071.4 files/s, 411121.5 lines/s)
------------------------------------------------------------------------------------
Language                          files          blank        comment           code
------------------------------------------------------------------------------------
Erlang                             1096          37380          32964         222644
JSON                                 57             65              0          31399
Elixir                              415           7958           2977          27974
JavaScript                           70           4493           4710          23345
Markdown                            250          12176             14          17150
Starlark                            129           1880           1812          17035
make                                 82           1957            759          10014
EJS                                  58            249              6           6218
Text                                 69           1191              0           5345
Java                                 27            884            427           4603
Python                               48           1073            929           4521
Bourne Shell                         64            646            629           3283
HTML                                 14            192              4           2985
YAML                                 21             47            138           2284
DOS Batch                            17            326            163           1390
Bourne Again Shell                   10            144              5            802
XML                                  17             80            300            770
Maven                                 7             91              9            743
AsciiDoc                              4            176              0            705
HCL                                   9            104             33            536
F#                                    1             71             12            476
C#                                   13             92             40            365
CSS                                   3             76              2            326
Dockerfile                            5             32             70            262
SVG                                   2              0              2            244
Ruby                                 18             64             19            217
Kotlin                                3             26             41            151
MSBuild script                        3              5              7            149
PHP                                   5             24             63            105
Bazel                                 1             31              0            103
Perl                                  6              9              4             69
Properties                            7             12              0             54
Visual Studio Solution                2              2              2             46
INI                                   3             12              0             44
CSV                                   1              2              0             24
diff                                  1              2              6              7
Groovy                                1              2              2              6
ASP.NET                               1              0              0              1
------------------------------------------------------------------------------------
SUM:                               2540          71574          46149         386395
------------------------------------------------------------------------------------

These might be overcounts for these projects, so I would only take these numbers as indicative that BlazingMQ is not significantly larger than other open-source message queues. There's simply a certain amount of inherent complexity that any enterprise-grade message queue must contain.

That said, if you're interested in learning more about BlazingMQ or perhaps contributing to the project, this does make it a bit intimidating. I would recommend looking at our Getting Started guide, our C++ library API docs, and our (work-in-progress) protocol guide. Additionally, you may also be interested in our Java SDK (and its API docs), which is much smaller, at the cost of reduced functionality exposed to users.

[billowqiu]

@pniedzielski Thank you for your wonderful reply.

[quarter-note]

@billowqiu Please feel free to reopen if you have any additional questions. Thanks!