keathley/twirp-elixir

Elixir implementation of the twirp RPC framework

Twirp

Twirp provides an Elixir implementation of the Twirp RPC framework developed by Twitch. The protocol defines semantics for routing and serialization of RPCs based on protobufs.

https://hexdocs.pm/twirp.

Installation

Add Twirp to your list of dependencies:

def deps do
  [
    {:twirp, "~> 0.8"}
  ]
end

If you want to be able to generate services and clients based on proto files you'll need the protoc compiler (brew install protobuf on MacOS) and both the twirp and elixir protobuf plugins:

$ mix escript.install hex protobuf
$ mix escript.install hex twirp

Both of these escripts will need to be in your $PATH in order for protoc to find them.

Example

The canonical Twirp example is a Haberdasher service. Here's the protobuf description for the service.

syntax = "proto3";

package example;

// Haberdasher service makes hats for clients.
service Haberdasher {
  // MakeHat produces a hat of mysterious, randomly-selected color!
  rpc MakeHat(Size) returns (Hat);
}

// Size of a Hat, in inches.
message Size {
  int32 inches = 1; // must be > 0
}

// A Hat is a piece of headwear made by a Haberdasher.
message Hat {
  int32 inches = 1;
  string color = 2; // anything but "invisible"
  string name = 3; // i.e. "bowler"
}

We'll assume for now that this proto file lives in priv/protos/service.proto

Code generation

We can now use protoc to generate the files we need. You can run this command from the root directory of your project.

$ protoc --proto_path=./priv/protos --elixir_out=./lib/example --twirp_elixir_out=./lib/example ./priv/protos/service.proto

After running this command there should be 2 files located in lib/example.

The message definitions:

defmodule Example.Size do
  @moduledoc false
  use Protobuf, syntax: :proto3

  @type t :: %__MODULE__{
          inches: integer
        }

  defstruct [:inches]

  field :inches, 1, type: :int32
end

defmodule Example.Hat do
  @moduledoc false
  use Protobuf, syntax: :proto3

  @type t :: %__MODULE__{
          inches: integer,
          color: String.t(),
          name: String.t()
        }
  defstruct [:inches, :color, :name]

  field :inches, 1, type: :int32
  field :color, 2, type: :string
  field :name, 3, type: :string
end

The service and client definition:

defmodule Example.HaberdasherService do
  @moduledoc false
  use Twirp.Service

  package "example"
  service "Haberdasher"

  rpc :MakeHat, Example.Size, Example.Hat, :make_hat
end

defmodule Example.HaberdasherClient do
  @moduledoc false
  # client implementation...
end

Implementing the server

Now that we've generated the service definition we can implement a "handler" module that will implement each "method".

defmodule Example.HaberdasherHandler do
  @colors ~w|white black brown red blue|
  @names ["bowler", "baseball cap", "top hat", "derby"]

  def make_hat(_ctx, size) do
    if size.inches <= 0 do
      Twirp.Error.invalid_argument("I can't make a hat that small!")
    else
      %Example.Hat{
        inches: size.inches,
        color: Enum.random(@colors),
        name: Enum.random(@names)
      }
    end
  end
end

Separating the service and handler like this may seem a little odd but there are good reasons to do this. The most important is that it allows the service to be autogenerated again in the future. The second reason is that it allows us to easily mock service implementations for testing.

Running the server

To serve traffic Twirp provides a Plug. We use this plug to attach our service definition with our handler.

defmodule Example.Router do
  use Plug.Router

  plug Twirp.Plug, service: Example.HaberdasherService, handler: Example.HaberdasherHandler
end

defmodule Example.Application do
  use Application

  def start(_type, _args) do
    children = [
      Plug.Cowboy.child_spec(scheme: :http, plug: Example.Router, options: [port: 4040]),
    ]

    opts = [strategy: :one_for_one, name: Example.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

If you start your application your plug will now be available on port 4040.

Using the client

Client definitions are generated alongside the service definition. This allows you to generate clients for your services in other applications. You can make RPC calls like so:

defmodule AnotherService.GetHats do
  alias Example.HaberdasherClient, as: Client
  alias Example.{Size, Hat}

  def make_a_hat(inches) do
    case Client.make_hat(Size.new(inches: inches)) do
      {:ok, %Hat{}=hat} ->
        hat

      {:error, %Twirp.Error{msg: msg}} ->
        Logger.error(msg)
    end
  end
end

Running with Phoenix

The plug can also be attached within a Phoenix Router. Example below would be accessible at /rpc/hat

URL: /rpc/hat/twirp/example.Haberdasher/MakeHat or /{prefix?}/twirp/{package}.{service}/{method}

defmodule ExampleWeb.Router do
  use ExampleWeb, :router

  scope "/rpc" do
    forward "/hat", Twirp.Plug,
      service: Example.HaberdasherService, handler: Example.HaberdasherHandler
  end
end

Client adapters

Twirp supports either Finch or Hackney as the underlying http client. Finch is the default. If you want to configure the adapter you can pass the adapter option when you call start_link.

Client.start_link(
  url: "https://some.url",
  adapter: {Twirp.Client.Hackney, [pool_opts: [timeout: 30_000, max_connections: 100]]}
)

Should I use this?

This implementation is still early but I believe that it should be ready for use. One of the benefits of twirp is that the implementation is quite easy to understand. At the moment there's only about ~600 LOC in the entire lib directory so if something goes wrong it shouldn't be hard to look through it and understand the issue.

twirp-elixir's Issues

Publishing clients to hex

We want to be able publish generated clients to our private hex registry. This presents some interesting challenges.

Ideally, we'd be able to publish the client directly from the service that defines it. But this isn't quite possible because the client has a dependence on twirp and the HTTP client. We can't publish the entire service mix.exs file since it would include all of the dependencies that the service is using and there might be conflicts between the provider service and the consumer service.

I'm not sure that there is a great way around this. The only thing I can think of is to create a separate mix project that contains the service definition and the client. Both the providing service and the consuming service would need to import it as a dependency. That seems annoying. But I'm not sure I have a better solution.

telemetry version conflicts in newer app

Getting errors importing twirp-elixir into applications that are using telemetry version 1.

Looking at how other libraries are handling this, it appears that supporting both .4 and 1.0 is a common pattern.
https://github.com/phoenixframework/phoenix/blob/master/mix.exs#L66
https://github.com/elixir-ecto/ecto/blob/master/mix.exs#L35

#119 -- follows the above approach

Implement Twirp protocol v7

Changes required to implement the new protocol:

• Error code resource_exhausted should map to 429 (instead of 403).
• Allow any arbitrary URL prefixes instead of forcing the "/twirp".

See Protocol v7 differences and the Go implementation v7.0.0 release notes for details.

Client generation should have no dependency on twirp itself

This caused an issue a while ago where the generated client depended on a more recent version of twirp. Based on experience, I think we want to generate the client in such a way as to have no dependency other than the http client.

(KeyError) key :elixir_module_prefix not found in: %Google.Protobuf.FileOptions{}

(KeyError) key :elixir_module_prefix not found in: %Google.Protobuf.FileOptions{}

https://github.com/keathley/twirp-elixir/blob/master/lib/twirp/protoc/generator.ex#L25 it appears protobuf-elixir/protobuf no longer has that key though it looks like it might exist as some sort of monkey-patched protobuf extension if you require the right module.

I'm sorry this isn't a patch, I'm new to Elixir and unsure if we should be testing for the key or requiring in the extension.

Support for `oneof` attributes missing when using JSON?

Thank you for this package!

Assume the following protobuf definition:

 message Result {
  oneof result {
    Success success = 1;    
    Error error = 2;          
  }

  message Success {
    string processed_at = 1;  
  }

  message Error {
    int32  status = 1;        
    string title = 2;         
    string detail = 3;        
  }
}

The elixir struct looks this:

defmodule Result do
  @moduledoc false
  use Protobuf, syntax: :proto3

  @type t :: %__MODULE__{
          result:
            {:success, Result.Success.t() | nil}
            | {:error, Result.Error.t() | nil}
        }

  defstruct result: nil

  oneof :result, 0

  field :success, 1, type: Result.Success, oneof: 0
  field :error, 2, type: Result.Error, oneof: 0
end

If I create new Result:

    success = Result.Success.new(processed_at: processed_at)
    Result.new(result: {:success, success})

An error is returned from the RPC if I use JSON as output format:

protocol Jason.Encoder not implemented for {:success,....

But if I use the Protobuf.JSON.to_encodable/1 from :protobuf it correct (lowerCamelCase):

%{"success" => %{"processedAt" => ...}}

Solution: Instead of using Jason to convert the generated structs, the function Protobuf.JSON.to_encodable/1 should used to convert the struct in a previous step.

Of course, it is possible to use the Twirp.Error stuff, but this is only an example, that in case of oneof definition the Jason encoder should not called direct on the structs, because the oneof attribute uses tuples like {:key, value} to support several options.

Read comments in protobuf and add them as docs to client functions