hbpvis/zeroeq

Cross-platform C++ library for fast binary and REST messaging

zeroeq's Issues

Handle zeroconf disconnects

Currently we only connect() newly found Publishers and Servers, but never disconnect them. Less important for pub-sub, but deadly for req-rep.

Remote connection broker

Use Case

A remote publisher needs a subscriber to subscribe to him directly, since ZeroConf is not available between the local and remote nodes.

Design

Two new classes in a nested zeq::connection namespace provide a Broker on the subscriber side and a Service on the publisher side. They are implemented internally using the request (service) - reply (broker) pattern. The broker has a Subscriber, to which it adds connections it receives from the service. The Service has a subscribe method taking a Publisher (address).

Implementation

• Broker/Service classes
• #30: New Receiver base class implementing the zmq_poll logic. Base class of subscriber and broker. Has a Receiver( Receiver& shared ) ctor to allow one receive() poll all receiver sockets.

Update cppnetlib subproject

Currently, you have cppnetlib @ cef6472 as a subproject of ZeroEQ.

That version of cppnetlib contains this libs/network/src/uri/schemes.cpp file:
https://github.com/cpp-netlib/cpp-netlib/blob/cef6472065a3d964b765681f5988e28866c26d74/libs/network/src/uri/schemes.cpp

But that file has two "defined but not used" warnings at lines 47 and 48, that prevent ZeroEQ to be compiled with the option -Werror=unused-variable, or any project that uses ZeroEQ as dependency.

I know that you are not responsible for this warnings, but could you please consider to upgrade the version of cppnetlib you are using?

git_external for ccp-netlib

Hi Guys,

I was upgrading the nix builds for ZeroEQ and I have seen that you sneak-in one git_external dependency.

git_external(${COMMON_SOURCE_DIR}/cppnetlib
https://github.com/BlueBrain/cpp-netlib 0.11-bbp)

Please, could you provide a way to disable that and build without git_external, by using a cppnetlib installation path manually provided through CMAKE_PREFIX_PATH like it works for subprojects.

Like I already said, no decent package manager would accept a software to download random staff during configure phase. This is a blocker for me to Upgrade ZeroEQ, and the upgrade is necessary for other package due to API changes.

Cheers,
Adrien

Increase robustness with broken name resolution

When using zeroconf browsing, clients are often not resolvable which causes the following exception:

52384.Main     rd/src/zeq/zeq/subscriber.cpp:54   25054 Cannot connect subscriber to tcp://dhcp587-146.laptop-wlc.uni-heidelberg.de:55113, got Invalid argument
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Cannot connect subscriber to tcp://dhcp587-146.laptop-wlc.uni-heidelberg.de:55113, got Invalid argument
Abort

Fix: catch and ignore?

Interactive Supercomputing video is private

Unable to view https://www.youtube.com/watch?v=wATHwvRFGz0&t=1m36s as it marked as private on youtube

.gitsubprojects does not reference to master

In order to pull the latest changes in the master ZeroBuf within zeq, I needed to change the .gitsubprojects to:
git_subproject(ZeroBuf https://github.com/HBPVIS/ZeroBuf.git master).
Is there a reason on why specific versions are pulled here instead of the most current source?

Ubuntu 16.04 compile error

Hey,
Just checked out ZeroEQ (required as a dependency for @favreau 's Brayns package), and get the following compile error(s) (using latest libzmq-dev package, installed right now just for this project):
(System Ubuntu 16.04, up to date with latest updates)

[ 44%] Building CXX object zeroeq/CMakeFiles/ZeroEQ.dir/connection/broker.cpp.o
In file included from /home/wald/Projects/ZeroEQ/zeroeq/connection/broker.cpp:9:0:
/home/wald/Projects/ZeroEQ/zeroeq/detail/sender.h: In constructor ‘zeroeq::detail::Sender::Sender(const zeroeq::URI&, void*, int)’:
/home/wald/Projects/ZeroEQ/zeroeq/detail/sender.h:43:33: error: ‘ZMQ_SNDHWM’ was not declared in this scope
zmq_setsockopt( socket, ZMQ_SNDHWM, &hwm, sizeof( hwm ));
^
/home/wald/Projects/ZeroEQ/zeroeq/detail/sender.h: In destructor ‘zeroeq::detail::Sender::~Sender()’:
/home/wald/Projects/ZeroEQ/zeroeq/detail/sender.h:51:39: error: ‘zmq_ctx_destroy’ was not declared in this scope
zmq_ctx_destroy( _context );
^
/home/wald/Projects/ZeroEQ/zeroeq/detail/sender.h: In member function ‘void zeroeq::detail::Sender::_getEndPoint(std::__cxx11::string&, std::__cxx11::string&) const’:
/home/wald/Projects/ZeroEQ/zeroeq/detail/sender.h:100:37: error: ‘ZMQ_LAST_ENDPOINT’ was not declared in this scope
if( zmq_getsockopt( socket, ZMQ_LAST_ENDPOINT, &buffer, &size ) == -1 )
^
/home/wald/Projects/ZeroEQ/zeroeq/detail/sender.h: In member function ‘void* zeroeq::detail::Sender::_createContext(void*)’:
/home/wald/Projects/ZeroEQ/zeroeq/detail/sender.h:125:32: error: ‘zmq_ctx_new’ was not declared in this scope
context = zmq_ctx_new();
^
/home/wald/Projects/ZeroEQ/zeroeq/connection/broker.cpp: In member function ‘void zeroeq::connection::detail::Broker::process(zeroeq::detail::Socket&)’:
/home/wald/Projects/ZeroEQ/zeroeq/connection/broker.cpp:67:47: error: ‘zmq_msg_recv’ was not declared in this scope
zmq_msg_recv( &msg, socket.socket, 0 );
^
compilation terminated due to -fmax-errors=5.
zeroeq/CMakeFiles/ZeroEQ.dir/build.make:86: recipe for target 'zeroeq/CMakeFiles/ZeroEQ.dir/connection/broker.cpp.o' failed
make[2]: *** [zeroeq/CMakeFiles/ZeroEQ.dir/connection/broker.cpp.o] Error 1
CMakeFiles/Makefile2:2530: recipe for target 'zeroeq/CMakeFiles/ZeroEQ.dir/all' failed
make[1]: *** [zeroeq/CMakeFiles/ZeroEQ.dir/all] Error 2
Makefile:160: recipe for target 'all' failed
make: *** [all] Error 2

Building Zeq on Intel cluster

Hello, I'm trying to build Zeq on an Intel cluster, but I'm getting the following error messages when building using make :

[  1%] Building CXX object Servus/servus/CMakeFiles/Servus.dir/version.cpp.o
[  3%] Building CXX object Servus/servus/CMakeFiles/Servus.dir/md5/md5.cc.o
[  4%] Building CXX object Servus/servus/CMakeFiles/Servus.dir/servus.cpp.o
~/externals/zeq/Servus/servus/servus.cpp(48): warning #2304: non-explicit constructor with single argument may cause implicit type conversion
      Servus( const std::string& name ) : _name( name ) {}
      ^

In file included from ~/externals/zeq/Servus/servus/servus.cpp(165):
~/externals/zeq/Servus/servus/none/servus.h(26): warning #2304: non-explicit constructor with single argument may cause implicit type conversion
      Servus( const std::string& name ) : detail::Servus( name ) {}
      ^

[  4%] Building CXX object Servus/servus/CMakeFiles/Servus.dir/uint128_t.cpp.o
~/externals/zeq/Servus/servus/uint128_t.cpp(86): error: namespace "std" has no member "mt19937_64"
          static std::mt19937_64 engine( device( ));
                      ^

~/externals/zeq/Servus/servus/uint128_t.cpp(87): error: qualified name is not allowed
          static std::uniform_int_distribution< uint64_t > generator(
                 ^

~/externals/zeq/Servus/servus/uint128_t.cpp(87): error: expected a ";"
          static std::uniform_int_distribution< uint64_t > generator(
                                              ^
~/externals/zeq/Servus/servus/uint128_t.cpp(98): error: identifier "generator" is undefined
          value.high() = generator( engine );
                         ^

compilation aborted for ~/externals/zeq/Servus/servus/uint128_t.cpp (code 2)
make[2]: *** [Servus/servus/CMakeFiles/Servus.dir/uint128_t.cpp.o] Error 2
make[1]: *** [Servus/servus/CMakeFiles/Servus.dir/all] Error 2
make: *** [all] Error 2

I checked on google and found out it may be a compatibility issue with the compilers.
The CXX environnement variable (which I assume is the compiler used here) is icpc version 14.0.3 (gcc version 4.4.7 compatibility).
Anyone has an idea how I could fix this problem ?

Thank you for your time

Implement HTTP Server

Specification: https://github.com/HBPVIS/zeq/blob/master/doc/feature/httpServer.md

Zeq,ZeroMQ DLLs crashing on Windows

Hello,
I'm currently trying to use Zeq for a client program along with Vista on Windows x64

This is the content of my CMakeList.txt :

# $Id:$

cmake_minimum_required( VERSION 2.8 )
project( Visualiser )

if( "${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU" OR "${CMAKE_CXX_COMPILER_ID}" STREQUAL "Intel" )

                set( CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11" CACHE STRING "Force gcc and intel compiler to compile it as c++11 code" FORCE )

endif( "${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU" OR "${CMAKE_CXX_COMPILER_ID}" STREQUAL "Intel" )

list( APPEND CMAKE_MODULE_PATH "$ENV{VISTA_CMAKE_COMMON}" )

include( VistaCommon )

vista_use_package( VistaCoreLibs "HEAD" REQUIRED FIND_DEPENDENCIES )

#find_package( ZEQ REQUIRED NO_CMAKE_BUILDS_PATH )
vista_use_package (ZEQ REQUIRED)
include_directories(${ZEQ_INCLUDE_DIRS})

# Including the source files of all source subfolders recursively
include( "src/_SourceFiles.cmake" )

add_executable( Visualiser ${ProjectSources} )
target_link_libraries( Visualiser
    ${VISTA_USE_PACKAGE_LIBRARIES} # contains all libraries from vista_use_package() calls
    zeq
    zeqHBP
    Servus
)

vista_configure_app( Visualiser )
vista_create_default_info_file( Visualiser )

I've got a simple Vista application, which works with this CMakeList.txt file as long as I don't try to use anything related to zeq.

Unfortunatly, when I include this line :

#include <zeq/zeq.h>

I get an error message when launching my program saying that it can't find "libzmq_d.dll".
I'm actually not sure why it's trying to load the _d one, but anyway, I tried copying it to the executable folder so that it won't have troubles finding it, and now it simply crashes before even starting to do anything.
Even with a completly empty main, it still crashes.

If you have any idea of what could be wrong, please tell me.

Thank you for your time.

Support paramters/queries for HTTP GET

Use case: GET ImageJPEG with size and compression quality

Future

• Implement #152
• Delay request-reply until more expertise or use cases are available
• Return HTTP error codes (enum'd) from SerializeHandler (eg request accepted [202]) (needs spec)

throw in handle functions not handled

in http::Server and the Server from #219, throws from the handlers should be handled and an error reported to the client.

http server may hang in receive() when remote connection is closed abruptly

The application deadlocks at ::zmq::send() at lines 195-197 of server.cpp trying to send a return code 200:

if( ::zmq_send( socket, id, idSize, ZMQ_SNDMORE ) != idSize ||
        ::zmq_send( socket, rep.c_str(), rep.length(), more ) !=
        int( rep.length( )))
{
ZEROEQWARN << "Could not send HTTP response header: "
                          << zmq_strerror( zmq_errno( )) << std::endl;
return;
}

Notify "empty" object

Current FBEvents with no payload have no notification (notify is called in fromBinary which is within if( payload).

Solution 1: Use event-based notification for these, and not objects (#152).
Solution 2: re-introduce some notifyFoo()?

Is bool Publisher::publish( const Event& event ) deprecated?

Hi,

In the doxygen documentation it says it is deprecated but it doesn't dump deprecation message through ZEQWARN as the servus::URI-based constructor does.

Thanks,
Pablo.

MicroServices capability?

Came across this library while looking for a good microservice platform for C/C++. This seems to be a good work, but trying to understand more about its capabilities. Any pointers / help is appreciated.

Some of the capabilities I am looking for are:

• Load balanced RPC providers (with multiple scheduling methods support, such as Round-Robin, LRU etc.)
• Dynamic providers that can come and go anytime (connected through, say, wifi hotspots or any narrowband networks)
• Polyglot providers (RPC methods created in various multiple languages, but exposed, with just a line or two code, to the framework as RPC providers)
• Mesh capability. Something similar to Seneca Mesh and/or interoperates with similar mesh networks based on patterns (thus creating complete transparency of the provider)

In the Industrial IOT field, these above are the need of the hour and ZeroEQ can be a good starting point to achieve these, since it already has REST and RPC, making it easy to achieve the Polyglot capability. If more architectural details are documented/explained, others like me would be happy to contribute where possible. I would love to see Load-balanced Polyglot RPC providers that dynamically gets discovered through Mesh and provide the services to the clients and go away.

HTTP Server freezes with multi-packet requests

In the Server::Impl::process method, the request.complete() returns true on when the first packet is received. If the request arrives in many packets, only the first one is considered, and the message is trucated. The error is then propagated to the JSON parser, that in the end freezes the HTTP interface.

I worked around the problem by ignoring such requests, but this is of course not acceptable.

      if( consumed != msgSize )
                    return; // incomplete message, ignore

I have the feeling that the problem is in the underlying http lib that is not properly setting the complete flag.

I fail to build the zeq

When I cmake the zeq as you said, it will arise the error "CMake Error at CMakeLists.txt:(flatbuffers_generate_c_headers): Unknown CMake command "flatbuffers_generate_c_headers" ". Can you tell me the wrong I'm with?