end2endzone/protobuf-pbop-plugin

protobuf-pbop-plugin is a C++ open-source plugin for Google Protocol Buffers which provides inter-process communication (IPC) over Windows Named Pipes. The acronym PBOP is for Protocol Buffers Over Pipe (PBOP)

protobuf-pbop-plugin

protobuf-pbop-plugin is a C++ open-source plugin for Google Protocol Buffers which provides inter-process communication (IPC) over Windows Named Pipes.

The acronym PBOP is for Protocol Buffers Over Pipe (PBOP)

Status

Build:

Service	Build	Tests
AppVeyor

Statistics:

Purpose

There is already multiple RPC options for implementing a client-server application. One of them is gRPC which provides remote procedure call (RPC) support over sockets. In 99% of use case scenarios where an IPC or RPC solution is required, gRPC is the way to go.

However, gRPC have limitations on Windows platform. One of them is performance when communicating locally. Sockets have a relatively high overhead when communicating locally between processes because messages must go through the network routing protocols layer. On Unix systems, one can use Unix domain socket for low latency local communication since they don't go through this layer. Yet on Windows, there is no support for domain sockets.

For local inter-process communication, in situations where low latency is a requirement, Named Pipes offer a great alternative over sockets. For examples, pipes are preferable over sockets in real-time application, for services that must process multiple queries in a short period.

protobuf-pbop-plugin was created for this purpose. It provides a simple solution for implementing low-latency, fast inter-process communication (IPC) solution over Named Pipes on Windows.

gRPC do not support Windows Named Pipes but a feature request is currently open.

Other developer have created a similar project but they don't seem to support protobuf services.

Features

The main features of protobuf-pbop-plugin are:

• Prebuild Windows Named Pipe Client and Server base classes.
• Event based programming for the Server class: Startup, Shutdown, Listening, Connection, Client-Created, Client-Disconnected, Client-Destroyed, Client-Error.
• Rich error support: all utility functions return a Status class which encapsulates an error code and error message.
• Build on top of Google's Protocol Buffers library. Get serialization performance and robustness from a tested source.

Note: protobuf-pbop-plugin does not support the following features:

• Client or Server streaming RPCs: service methods that uses or returns a stream instead of a message.
• Asynchronous IPC calls.

Usage

The following instructions show how to use protobuf-pbop-plugin.

About Protocol Buffers

protobuf-pbop-plugin is a code generator plugin for Protocol Buffers which is Google's library for serializing structured data. Once data is serialized, it can be transfered from a client to a server (or from a server back to a client).

Before continuing with the plugin, a good understanding of how to use protocol buffers is required. If you are already familiar with Protocol Buffers, you can skip this section.

If not, a good starting point is the official c++ tutorial.

Another good resource is the section Working with Protocol Buffers in gRPC documentation. Note that gRPC is also a Protocol Buffers plugin.

About the generated code

When invoked, the plugin will generate c++ code for each service defined in your proto file.

For each service, a class matching the service name will be generated with the following :

1. An interface called StubInterface that defines all the service methods.
2. A Client class that implements the service interface and have all the required functionality to establish and handle a pipe connection to a pbop::Server.
3. A Service class that implements the StubInterface and pbop::Service interface. This service class can be assigned to a pbop::Server which listens for incoming pipe connections.

See the example section for details.

The generated code have a dependency on the following libraries:

• protobuf-pbop-plugin
• Google's Protocol Buffers (protobuf)
• zLib

Command line execution

On Windows, the compiler executable filename is protoc.exe. This executable's directory must be in the system's PATH environment variable to be used properly.

A Protocol Buffers plugin can be called by two different ways:

1. From the PATH environment variable.
2. From an absolute path.

This section explains the details of each method.

Using PATH environment variable

This method is helpful if the plugin executable is in PATH environment variable.

The following command line, will execute the plugin and generate the required code:

protoc.exe --<plugin_filename_executable>_out=<output_directory> --proto_path=<protobuf_include_directory>;<target_proto_directory>; <target_proto_filename>

where

• <plugin_filename_executable> matches the filename without extension of the plugin executable.
• <output_directory> matches the output directory of the generated code.
• <protobuf_include_directory> matches the include directory of the protobuf library.
• <target_proto_directory> matches the directory of the target proto file.
• <target_proto_filename> matches the path to the target proto file.

For example:

protoc.exe --protobuf-pbop-plugin_out=C:\Projets\demoplugin\output --proto_path=C:/Projets/third_parties/protobuf/install/include;C:\Projets\demoplugin\protos; C:\Projets\demoplugin\protos\demo.proto

Note: This method is working because the plugin's filename is protobuf-pbop-plugin.exe.

Using plugin's absolute path

This method shall be used if the plugin's executable is not in PATH environment variable. The absolute path to the executable must be specified in the command line.

The following command line, will execute the plugin and generate the required code:

protoc.exe --plugin=protoc-gen-<plugin_short_name>=<plugin_executable_path> --<plugin_short_name>_out=<output_directory> --proto_path=<protobuf_include_directory>;<target_proto_directory>; <target_proto_filename>

where

• <plugin_short_name> matches a unique identifier for this plugin. Can be any word.
• <plugin_executable_path> matches the plugin's executable path.
• <output_directory> matches the output directory of the generated code.
• <protobuf_include_directory> matches the include directory of the protobuf library.
• <target_proto_directory> matches the directory of the target proto file.
• <target_proto_filename> matches the path to the target proto file.

For example:

protoc.exe --plugin=protoc-gen-pbop=C:\Projets\demoplugin\bin\protobuf-pbop-plugin.exe --pbop_out=C:\Projets\demoplugin\output --proto_path=C:/Projets/third_parties/protobuf/install/include;C:\Projets\demoplugin\protos; C:\Projets\demoplugin\protos\demo.proto

Note: In the previous example, the identifier pbop is used as the plugin's short name.

Example: Greetings service

The following section show an actual example of using protobuf-pbop-plugin.

The greetings.proto file defines a single service called Greeter. The service has two service methods: SayHello and SayGoodbye. Each service method has their own "request" and "response" set of messages.

greetings.proto file

syntax = "proto3";

package greetings;

// The SayHello request message
message SayHelloRequest {
  string name = 1;
}

// The SayHello response message
message SayHelloResponse {
  string message = 1;
}

// The SayGoodbye request message
message SayGoodbyeRequest {
  string name = 1;
}

// The SayGoodbye response message
message SayGoodbyeResponse {
  string message = 1;
}

// The greeting service definition.
service Greeter {
  // Sends a greeting
  rpc SayHello (SayHelloRequest) returns (SayHelloResponse);
  
  // Sends a goodbye
  rpc SayGoodbye (SayGoodbyeRequest) returns (SayGoodbyeResponse);
}

Creating a client

With the following, one can create a client that creates a pipe connection to a server:

#include <stdio.h>
#include "greetings.pb.h"
#include "greetings.pbop.pb.h"
#include "pbop/Server.h"
#include "pbop/PipeConnection.h"

static const char * kPipeName = "\\\\.\\pipe\\greetings.pipe";

int main(int argc, char* argv[])
{
  printf("Running %s...\n", argv[0]);

  pbop::PipeConnection * connection = new pbop::PipeConnection();
  pbop::Status status = connection->Connect(kPipeName);
  if (!status.Success())
  {
    printf("Error in main(): %d, %s\n", status.GetCode(), status.GetDescription().c_str());
    return status.GetCode();
  }

  greetings::Greeter::Client client(connection);

  greetings::SayHelloRequest request;
  greetings::SayHelloResponse response;

  request.set_name("bob");

  printf("%s says hello.\n", request.name().c_str());
  status = client.SayHello(request, response);
  if (!status.Success())
  {
    printf("Error in main(): %d, %s\n", status.GetCode(), status.GetDescription().c_str());
    return status.GetCode();
  }

  printf("Message from server: %s\n", response.message().c_str());

  return 0;
}

Creating a server

With the following, one can create a server that listens for a connection from a client:

#include <stdio.h>
#include "greetings.pb.h"
#include "greetings.pbop.pb.h"
#include "pbop/Server.h"

static const char * kPipeName = "\\\\.\\pipe\\greetings.pipe";

class GreeterServiceImpl : public greetings::Greeter::Service
{
public:
  GreeterServiceImpl() {}
  virtual ~GreeterServiceImpl() {}

  pbop::Status SayHello(const greetings::SayHelloRequest & request, greetings::SayHelloResponse & response)
  {
    response.set_message("Greetings " + request.name());
    return pbop::Status::OK;
  }

  pbop::Status SayGoodbye(const greetings::SayGoodbyeRequest & request, greetings::SayGoodbyeResponse & response)
  {
    response.set_message("Farewell " + request.name());
    return pbop::Status::OK;
  }
};

int main(int argc, char* argv[])
{
  printf("Running %s...\n", argv[0]);

  GreeterServiceImpl * impl = new GreeterServiceImpl();

  pbop::Server server;
  server.RegisterService(impl);
  pbop::Status status = server.Run(kPipeName);
  if (!status.Success())
  {
    printf("Error in main(): %d, %s\n", status.GetCode(), status.GetDescription().c_str());
    return status.GetCode();
  }
  return 0;
}

Build

Please refer to file INSTALL.md for details on how installing/building the application.

Platform

protobuf-pbop-plugin has been tested with the following platform:

• Windows x86/x64

Versioning

This project use Semantic Versioning 2.0.0 for versioning. For the versions available, see the tags on this repository.

Authors

• Antoine Beauchamp - Initial work - end2endzone

See also the list of contributors who participated in this project.

License

This project is licensed under the MIT License - see the LICENSE file for details.

protobuf-pbop-plugin's Issues

Move the pipe listening code from Server::Run() to PipeConnection class

The recommended name for the function should be the following:

static Status PipeConnection::Listen(const char * pipe_name, PipeConnection ** connection);

When an incoming pipe connection is received by PipeConnection::Listen(), the function should create a new PipeConnection instance with the assigned pipe HANDLE.

Generate .cc files instead of .cpp

The plugin should generate a *.h file and a *.cc file to match the behavior of the default protocol buffers plugin for c++.

Evaluate the possibility of using a Thread class

The code might be more maintainable if a generic Thread class would be used instead of direclty using the win32 API.

Possible candidates:

• https://stackoverflow.com/questions/1372967/how-do-you-use-createthread-for-functions-which-are-class-members
• https://www.codeproject.com/Articles/21114/Creating-a-C-Thread-Class

Generated Client classes should take a pipe name in ctor instead of an actual PipeConnection.

The generated Client class should take a pipe name in ctor instead of an actual PipeConnection.
This would reduce the complexity of users to have to deal with PipeConnection instances and ownership.

Move PipeConnection::Connect() to a static function to be used as a factory method.

This is an optimization like PipeConnection::Listen(). It also reduce the complexity of handling the "cleanup" when calling Connect() on an object that is already connected. This method forces to deal with a new object every time. Once the object is initialized with a connected state, it can only go to a disconnected state and will stay like this until it is destroyed.

Also, consider both functions to return a Connection pointer instead of a PipeConnection.

Create more unit tests

The following tests should be implemented:

• A performance test that creates a Server and 5 Client instances. Verify how many queries can be executed in a given time.
• Unit test for the Status class.
• Unit test for the Client class.
• Unit test for the Server class.
• Unit test for handling two method calls at the same time. Create 1 call that is slow to process and one call that is fast. Try to execute the second call while processing the first call. There should be 1 client per thread but the server should be able to process calls from multiple threads at the same time.
• Unit test for validating errors propagation. Create a fake Service that returns errors to the client.
• Create unit test for how to handle errors when a client make a connection but the server is not running/responding.

Missing code documentation

The code in the pbop library and the generated code have no documentation.

Refactor Service::InvokeMethod() to use a string instead of an index for method identification

Using an index may not be ideal for searching for a method without description. A string which matches the function name would be much better.

For performance reason, a pointer to method may be required. See the following for implementation details:

• https://www.tutorialspoint.com/function-pointer-to-member-function-in-cplusplus#:~:text=In%20C%2B%2B%20%2C%20function%20pointers%20when,as%20they%20are%20type%20safe.
• https://isocpp.org/wiki/faq/pointers-to-members

Allow server shutdown when processing a service method call

The current code does not support shutting down the server when

A typical use case would be to have a service method (for example Shutdown()) which would call Server::Shutdown(). On shutdown, the main server thread would exit and the server program would exit gracefully.

Server blocking and unblocking modes

The server should be able to be used in blocking and non-blocking modes.

The following new api may be required:

• Start(), starts the server asynchronously.
• Wait() waits for the Shutdown() call. This functions blocks the execution until the shutdown signal is received.
• Shutdown(), stop listening for new connections. Stop existing connections form listening for new requests. Gracefully shutdown the server.

Asynchronous Named Pipes and IO Completion Ports

Evaluate the possibility of using Asynchronous Named Pipes and IO Completion Ports instead of blocking named pipes.

Create cmake script for running a *.proto file with the plugin.

The script should be able to extract the following information:

• Location of the protobuf compiler executable
• Location of the pbop plugin executable
• The output filenames from the input proto files

The argument to the script should be a list of *.proto files and a destination directory.

Create server callback for event based programming

Callback should allow to define custom event response on the following events:

• Server ready
• Server shutdown
• New connection
• Client disconnected

Missing documentation

A README file should be created to explain how to use the library.

Set server custom buffer size

The maximum size of a message transmitted through the pipe connection is currently hard coded to 4096 bytes. This value should be a variable to allow flexibility.

Use flags for ListenOptions to allow providing a single option without having to specify all options every time.

The PipeConnection::ListenOption structure should have a flag member to specify which options is specified.

Note: At the time of writing, only a single option buffer_size is available but more will be added eventually.

Define a method for shutting down the server

The shutdown method should be created in the Server class.

The shutdown shall also be made possible when processing a service method.

Rename Status::GetMessage() method to prevent conflicts with WIN32 api

The function GetMessage() is already reserved by the WIN32 api. This creates conflicts with the code.

Complete renaming product to pbop

There is ambiguity in the name of the product.

The repository is named protobuf-pipe-plugin.
The CMake project is called protobufpipeplugin.
The c++ namespace is pbop.

To remove any ambiguity, the following should be renamed as follows:

The repository name protobuf-pbop-plugin.
The CMake project pbop.
The c++ namespace pbop.

Rename `libProtobufPipePlugin` library

Find a shorter name for the library. The namespace of the library should also be changed to match the new name of the library.

Code refactoring

The following modifications to the code should be implemented:

• Split Connection interface from PipeConnection implementation class. Also add virtual destructor to Connection interface.
• Consider using a real HANDLE instead of size_t. This would force the usage of #include <Windows.h>. This would affect PipeConnection and Server classes. A cross-platform implementation would wrap the #include <Windows.h> inside a #ifdef _WIN32 statement.
• Create a Client class that would be stored in pbop library. This would reduce the size of generated client code and allow unit tests that connects a Client to Server.
• Reorder methods of Server class.
• The pbop::Service interface should only use base types.
• Refactor the generated code to match the following: a Protobuf service should not generate a namespace but a class with the service name. In the generated class, a StubInterface should be created (instead of an interface matching the service's name). A ServerStub should also be created within the service's class. Documentation should also be updated.