verizon/remotely

An elegant RPC system for reasonable people

Remotely

Remotely is a purely functional remoting library for reasonable people. It's fast, lightweight and models network operations as explicit monadic computations.

Please see the documentation for more information.

remotely's Issues

Provide transport layer versioning

From a long-term evolution perspective, have the ability to version and evolve the wire transport separate to the serialisation system being used for the 1-n frames for the payload blob. Sketch of this might be:

Frame 0: delimiting start frame
Frame 1: versioning / envelope data; probably a simplistic string
Frame 2-n: serialised blob, with bytes encoded with whatever system denoted by the envelope
Frame n+1: delimiting end frame

Fix Travis build flakiness

The Travis builds are failing because we never manage to pass the tests on all platforms. There does not seem to be any rhyme or reason about which platforms are failing so I am guessing it's some general flakiness, that may or may not be introduced by the fact that they are running in parallel.

In the mean time, it might be worth reducing the number of platforms to a single one and see if that improves things. A travis build that always fails is not very useful...

Codecs that don't align at byte boundaries are unsupported?

If I define a codec that doesn't align at byte boundaries, remotely doesn't like it. Here's a small reproducer: https://gist.github.com/larsrh/2d2480b3870c1165e9104225e14b94bc

According to @mpilquist this is in fact a valid codec. If I change bool to bool(8), it is aligned at byte boundaries and works as expected.

Quick digging into the sources revealed the following lines in remotely:

package object codecs extends lowerprioritycodecs {
  // [...]
  implicit val bool = C.bool(8) // use a full byte

This line has been there for at least two years, but I couldn't find any explanation anywhere. Could remotely not pad, or at least throw an exception explaining that the codec needs to be byte-aligned?

Finally, here's the stack trace:

$ sbt run
[info] Compiling 2 Scala sources to /home/lars/tmp/remotely-bug/target/scala-2.11/classes...
[info] Running Main 
[server] NEGOTIATION - got ssl parameters: None with localhost/127.0.0.1:8083
[server] NEGOTIATION - about to bind with localhost/127.0.0.1:8083
[server] NEGOTIATION - bound with localhost/127.0.0.1:8083
[server] NEGOTIATION - channel connected with /127.0.0.1:38980
[server] NEGOTIATION - sending capabilities with /127.0.0.1:38980
[server] NEGOTIATION - sent capabilities with /127.0.0.1:38980
[server] NEGOTIATION - creating queue with /127.0.0.1:38980
[server] NEGOTIATION - closing queue with /127.0.0.1:38980
[server] ----------------
[server] header: Map()
[server] trace: ac746630-5636-4905-9f07-f9d88a1d2b7a
[server] request: id(5)
[server] result: \/-(Elem((tag,List((type,int))),List(Elem((text,List((content,5))),List()))))
[server] duration: 20253373 nanoseconds
[client] ----------------
[client] header: Map()
[client] trace: ac746630-5636-4905-9f07-f9d88a1d2b7a
[client] request: id(5)
[client] result: -\/(java.lang.IllegalArgumentException: 6 bits remaining: 0x00)
[client] duration: 231969252 nanoseconds
[error] (run-main-3) java.lang.IllegalArgumentException: 6 bits remaining: 0x00
java.lang.IllegalArgumentException: 6 bits remaining: 0x00
        at scodec.interop.scalaz.package$AttemptSyntax$$anonfun$toTask$extension0$1.apply(package.scala:27)
        at scodec.interop.scalaz.package$AttemptSyntax$$anonfun$toTask$extension0$1.apply(package.scala:27)
        at scodec.interop.scalaz.package$AttemptSyntax$$anonfun$toTask$extension1$1.apply(package.scala:31)
        at scodec.interop.scalaz.package$AttemptSyntax$$anonfun$toTask$extension1$1.apply(package.scala:31)
        at scodec.Attempt$Failure.fold(Attempt.scala:113)
        at scodec.interop.scalaz.package$AttemptSyntax$.toTask$extension1(package.scala:31)
        at scodec.interop.scalaz.package$AttemptSyntax$.toTask$extension0(package.scala:27)
        at remotely.package$$anonfun$evaluate$1$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(package.scala:76)
        at remotely.package$$anonfun$evaluate$1$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(package.scala:70)
        at scalaz.concurrent.Task$$anonfun$flatMap$1$$anonfun$1.apply(Task.scala:36)
        at scalaz.concurrent.Task$$anonfun$flatMap$1$$anonfun$1.apply(Task.scala:36)
        at scalaz.concurrent.Task$.Try(Task.scala:389)
        at scalaz.concurrent.Task$$anonfun$flatMap$1.apply(Task.scala:36)
        at scalaz.concurrent.Task$$anonfun$flatMap$1.apply(Task.scala:34)
        at scala.Function1$$anonfun$andThen$1.apply(Function1.scala:52)
        at scalaz.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:58)
        at scalaz.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:58)
        at scalaz.concurrent.Future.step(Future.scala:109)
        at scalaz.concurrent.Future.listen(Future.scala:75)
        at scalaz.concurrent.Future$$anonfun$listen$1$$anonfun$apply$4.apply(Future.scala:79)
        at scalaz.concurrent.Future$$anonfun$listen$1$$anonfun$apply$4.apply(Future.scala:79)
        at scalaz.Free$$anonfun$map$1.apply(Free.scala:52)
        at scalaz.Free$$anonfun$map$1.apply(Free.scala:52)
        at scalaz.Free.scalaz$Free$$fastFlatMap(Free.scala:71)
        at scalaz.Free$$anonfun$resume$1.apply(Free.scala:87)
        at scalaz.Free$$anonfun$resume$1.apply(Free.scala:87)
        at scalaz.std.FunctionInstances$$anon$1$$anonfun$map$1.apply(Function.scala:56)
        at scalaz.Free$$anonfun$run$1.apply(Free.scala:186)
        at scalaz.Free$$anonfun$run$1.apply(Free.scala:186)
        at scalaz.Free.go2$1(Free.scala:133)
        at scalaz.Free.go(Free.scala:136)
        at scalaz.Free.run(Free.scala:186)
        at scalaz.concurrent.Future$$anonfun$async$1$$anonfun$apply$14.apply(Future.scala:376)
        at scalaz.concurrent.Future$$anonfun$async$1$$anonfun$apply$14.apply(Future.scala:376)
        at scalaz.stream.async.mutable.Queue$$anonfun$1.apply(Queue.scala:222)
        at scalaz.stream.async.mutable.Queue$$anonfun$1.apply(Queue.scala:221)
        at scalaz.stream.async.mutable.Queue$$anonfun$scalaz$stream$async$mutable$Queue$$stop$1$4$$anonfun$apply$8.apply$mcV$sp(Queue.scala:282)
        at scalaz.stream.async.mutable.Queue$$anonfun$scalaz$stream$async$mutable$Queue$$stop$1$4$$anonfun$apply$8.apply(Queue.scala:282)
        at scalaz.stream.async.mutable.Queue$$anonfun$scalaz$stream$async$mutable$Queue$$stop$1$4$$anonfun$apply$8.apply(Queue.scala:282)
        at scalaz.concurrent.StrategysLow$$anon$3$$anon$4.call(Strategy.scala:79)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
        at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 2 s, completed 21.11.2016 22:11:05

Issue with macros

So, I ran into a few issues with the macro stuff. First, the doc needs to mention that projects need to add the paradise plugin, otherwise the macro will fail silently and the eventual reference to .environment on the server impl will cause a compilation failure.

addCompilerPlugin("org.scalamacros" % "paradise" % "2.0.1" cross CrossVersion.full)

Next, I was unable to get the macro working if I factored out the protocol. So the following works:

@GenServer(remotely.Protocol
    .empty
    .codec[String]
    .codec[scalaz.\/[String, Int]] // fqn red'd even though it's imported
    .specify1[String, scalaz.\/[String, Int]]("pop")) 
abstract class World2Server

but if I try to factor out the protocol thus:

package demo

object WorldProto {
  val definition = remotely.Protocol
    .empty
    .codec[String]
    .codec[scalaz.\/[String, Int]]
    .specify1[String, scalaz.\/[String, Int]]("pop")
}

@GenServer(demo.WorldProto.definition) 
abstract class World2Server

compilation fails with the following message:

/Users/rnorris/Scala/doobie-remotely/src/main/scala/demo/World2.scala:26: exception during macro expansion: 
scala.tools.reflect.ToolBoxError: reflective compilation has failed: 
object definition is not a member of package demo.WorldProto
  at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.throwIfErrors(ToolBoxFactory.scala:315)
  at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.compile(ToolBoxFactory.scala:249)
  at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl.compile(ToolBoxFactory.scala:416)
  at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl.eval(ToolBoxFactory.scala:419)
  at scala.reflect.macros.runtime.Evals$class.eval(Evals.scala:16)
  at scala.reflect.macros.runtime.Context.eval(Context.scala:6)
  at remotely.GenServer$.impl(GenServer.scala:57)
@GenServer(demo.WorldProto.definition) 
 ^

I tried putting the protocol in another compilation unit and even in another project, but it didn't make a difference. I avoid macros so I really don't know what I should expect here; hopefully it's something dumb.

Upgrade to version 0.8.x of `scalaz-stream` and delete our own `uncons` implementation

The latest version of scalaz-stream contains an implementation of uncons that is generalized to any Monad. It also contains a bug fix over the implementation found within the Remotely code base.

Implement a 0mq transport layer

Implement a 0mq transport that uses the REQ & REP socket types. The intention here is that by making use of the already well-defined 0mq wire framing protocol, we can layer on application protocol versioning with relative ease. This also makes the integration path for non-scala services much smoother as they simply need to reuse an existing 0mq client in their language and use the primitives provided in a way thats compatible with the service.

Improve typesafety of Remote

Currently, it is possible to create any kind of Remote. This is not too much of an issue because remotes should be generated by the macro most of the time. It might still be interesting to associate the creation of a Remote with it's environment and/or protocol in order to make manual creation more typesafe. It would also in theory make the macro implementation more fool proof (in theory).

It might also allow to more easily assert the fact that it is only possible to stream one stream per Remote call (if we decide to go down that route).

Relying on the order of ScalaCheck property evaluation is unsafe

RemoteSpec relies on the order of ScalaCheck property evaluation in order to clean up the pool in the end. Here's what happens after upgrading to ScalaCheck 1.13.x:

[info] + Remote.roundtrip[List[Int]]: OK, passed 100 tests.
[info] Elapsed time: 0.630 sec 
[info] + Remote.check-serializers: OK, proved property.
[info] Elapsed time: 0.000 sec 
[info] + Remote.add3: OK, passed 100 tests.
[info] Elapsed time: 0.214 sec 
[info] + Remote.roundtrip: OK, passed 100 tests.
[info] Elapsed time: 0.307 sec 
[info] + Remote.check-declarations: OK, proved property.
[info] Elapsed time: 0.000 sec 
[info] + Remote.cleanup: OK, proved property.
[info] Elapsed time: 0.002 sec 
[info] ! Remote.roundtrip[Double]: Exception raised on property evaluation.
[info] > ARG_0: List()
[info] > ARG_1: Map()
[info] > Exception: java.lang.IllegalStateException: Pool not open
[info] org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
[info] org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
[info] org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
[info] remotely.transport.netty.NettyTransport$$anonfun$1.apply(Client.scala:36)
[info] remotely.transport.netty.NettyTransport$$anonfun$1.apply(Client.scala:35)
(more stack trace ...)

This blocks an upgrade to ScalaCheck 1.13.x. It can be easily fixed by converting it into a ScalaTest suite. See larsrh/remotely@30b0ae6 for an implementation.

support for scala 2.11.x

helpful stuff from @ahjohannessen on gitter;

Should resetAllAttrs not be replaced by resetLocalAttrs in GenServer and GenClient -- this is recommended by http://docs.scala-lang.org/overviews/macros/changelog211.html#how-to-make-your-210x-macros-work-in-2110

I am trying to build remotely with Scala 2.11.5 here - I get things to compile after replacing resetAllAttrs, though there are some deprecation warnings.

Any reason for being on 2.10.4?

Btw, here is an example of using macros with both 2.10.4 and 2.11.5: https://github.com/scalamacros/sbt-example/blob/master/project/Build.scala

Signatures 'wrapResponse' returns invalid type parameter string

Signatures.wrapResponse used by GenServer.scala returns invalid 'type parameter string' for list of parameters. There are missing parentheses around function parameters.
There is an error in SignaturesSpec.scala:

Signature("foo", List(Field("baz", "Baz"), Field("qux", "Qux")), "Zod").wrapResponse should be ("Baz,Qux => Response[Zod]")

should be:

Signature("foo", List(Field("baz", "Baz"), Field("qux", "Qux")), "Zod").wrapResponse should be ("(Baz,Qux) => Response[Zod]")

This causes error for protocols that use specify2.
Following protocol definition:

val definition = Protocol.empty
    .codec[Int]
    .specify2("f", Field.strict[Int]("a"), Field.strict[Int]("b"), Type[Int])

causes error during macro expansion:

[error] <toolbox>:1: wrong number of type parameters for method ref: [A](s: String)(implicit evidence$1: reflect.runtime.universe.TypeTag[A])remotely.Remote[A]
[error] val f = Remote.ref[Int,Int => Int]("f")

I can send two files to place in 'test' and 'test-server' modules that cause compile error seen above.

@GenClient cannot expand method named in reserved word properly

val definition = Protocol.empty.codec[String].specify1("type", Field.strict[String]("foo"), Type[String])

@GenClient(definition)
object FooClient

this throws compile error at
https://github.com/oncue/remotely/blob/master/core/src/main/scala/GenClient.scala#L46

since type is a reserved word.

easiest solution could be using backtick to surround the variable name, like this,

  val signatures : Set[Tree] = s.signatures.map { sig =>
      c.parse(s"""val `${sig.name}` = Remote.ref[${sig.typeString}]("${sig.name}")""")
  }

Provide transport or payload security

0mq 4 provides Curve based security; assess this and investigate if wrapping / proxying the connection via PKCS-based TLS would be in any way advantageous over Curve, or instead using 0mq 3.

error in macro expansion of @GenServer

// project 1
val definition = 
  Protocol
  .empty
  .codec[String]
  .codec[Option[String]]
  .codec[List[String]]    
  .codec[Map[String, String]]
  .specify3("foo", Field.strict[String]("a"), Field.strict[String]("b"), Field.strict[String]("c"), Type[List[String]])

// project 2 depends on project 1
// I did use the full fqn of definition
@GenServer(definition) abstract class Foo

It throws exception,

[error] scala.reflect.macros.ParseException: ';' expected but ',' found.
[error]         at scala.reflect.macros.contexts.Parsers$$anonfun$parse$1.apply(Parsers.scala:18)
[error]         at scala.reflect.macros.contexts.Parsers$$anonfun$parse$1.apply(Parsers.scala:17)
[error]         at scala.collection.mutable.LinkedHashSet.foreach(LinkedHashSet.scala:91)
[error]         at scala.reflect.macros.contexts.Parsers$class.parse(Parsers.scala:17)
[error]         at scala.reflect.macros.contexts.Context.parse(Context.scala:6)
[error]         at scala.reflect.macros.contexts.Context.parse(Context.scala:6)
[error]         at remotely.GenServer$.parseType(GenServer.scala:35)
[error]         at remotely.GenServer$$anonfun$2.apply(GenServer.scala:59)
[error]         at remotely.GenServer$$anonfun$2.apply(GenServer.scala:58)
[error]         at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
[error]         at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
[error]         at scala.collection.immutable.Set$Set2.foreach(Set.scala:111)
[error]         at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
[error]         at scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
[error]         at scala.collection.SetLike$class.map(SetLike.scala:92)
[error]         at scala.collection.AbstractSet.map(Set.scala:47)
[error]         at remotely.GenServer$.impl(GenServer.scala:58)
[error] @GenServer(....definition

I tried a few specify1, and they are all good though.

Generate certs from sbt so to avoid having to check them in

Also, in three years the current ones expire, at which time someone will spend an afternoon figuring out why the tests are suddenly failing (been there done that).

revert hlist addition to codecs

I think addition of hlists to Codecs has been more pain than anything else, compile times are staggering once your protocol grows. Honestly I do not see what value it brings about. This is the commit that introduced hlists on Codecs:

a087b00

@timperrett @stew @runarorama WDYT?

If reverting that change is is not desirable, could we then at least find a better way that does not make scalac crap out once the protocol starts to grow?

Fix "reject non auth clients when auth is required" test in SSLSpec

It's a copy paste from another test and does not test what it is supposed to based from the description

Allow for environments to be added

@ahjohannessen made a request in gitter that I think makes sense, which is that we could perhaps treat Environments as a monoid, so that they can be added together, and we can do something like:

(env1 |+| env2).serve(addr) ...

This would allow for remote functions to be logically grouped, and the later composed.

Release documentation website

It's out of date

Make `Protocol` and `Environment` more strongly typed

Probably using HLists to encode what codecs are part of a given Environment at the typelevel.

Ad-hoc testing tool for communicating with an arbitrary remotely service

During development and debugging it is nearly always needed to do some ad-hoc requests to a given system. Given remotely is using strongly typed contracts, we must devise a way of dynamically generating a client-side understanding of the server shapes, and be able to construct a set of input devices based on those shapes.

creating a transport is side-effecty

@tpolecat points out that NettyTransport.single(addr) is side-effecty and should probably return a Task. I tend to agree, but it might gross up the client code a bit. Open the issue for discussion before I go making this change.

Example using remotely?

Could you please point me to an example repo using remotely?

Thanks

investigate using netty4

We originally chose our netty version to align with akka, but we can perhaps have both netty 3 and netty 4 on the classpath without conflicting because they use different package names.