Remotely is a purely functional remoting library for reasonable people. It's fast, lightweight and models network operations as explicit monadic computations.
Please see the documentation for more information.
An elegant RPC system for reasonable people
Home Page: http://verizon.github.io/remotely/
License: Apache License 2.0
Remotely is a purely functional remoting library for reasonable people. It's fast, lightweight and models network operations as explicit monadic computations.
Please see the documentation for more information.
From a long-term evolution perspective, have the ability to version and evolve the wire transport separate to the serialisation system being used for the 1-n frames for the payload blob. Sketch of this might be:
Frame 0: delimiting start frame
Frame 1: versioning / envelope data; probably a simplistic string
Frame 2-n: serialised blob, with bytes encoded with whatever system denoted by the envelope
Frame n+1: delimiting end frame
The Travis builds are failing because we never manage to pass the tests on all platforms. There does not seem to be any rhyme or reason about which platforms are failing so I am guessing it's some general flakiness, that may or may not be introduced by the fact that they are running in parallel.
In the mean time, it might be worth reducing the number of platforms to a single one and see if that improves things. A travis build that always fails is not very useful...
If I define a codec that doesn't align at byte boundaries, remotely doesn't like it. Here's a small reproducer: https://gist.github.com/larsrh/2d2480b3870c1165e9104225e14b94bc
According to @mpilquist this is in fact a valid codec. If I change bool
to bool(8)
, it is aligned at byte boundaries and works as expected.
Quick digging into the sources revealed the following lines in remotely:
package object codecs extends lowerprioritycodecs {
// [...]
implicit val bool = C.bool(8) // use a full byte
This line has been there for at least two years, but I couldn't find any explanation anywhere. Could remotely not pad, or at least throw an exception explaining that the codec needs to be byte-aligned?
Finally, here's the stack trace:
$ sbt run
[info] Compiling 2 Scala sources to /home/lars/tmp/remotely-bug/target/scala-2.11/classes...
[info] Running Main
[server] NEGOTIATION - got ssl parameters: None with localhost/127.0.0.1:8083
[server] NEGOTIATION - about to bind with localhost/127.0.0.1:8083
[server] NEGOTIATION - bound with localhost/127.0.0.1:8083
[server] NEGOTIATION - channel connected with /127.0.0.1:38980
[server] NEGOTIATION - sending capabilities with /127.0.0.1:38980
[server] NEGOTIATION - sent capabilities with /127.0.0.1:38980
[server] NEGOTIATION - creating queue with /127.0.0.1:38980
[server] NEGOTIATION - closing queue with /127.0.0.1:38980
[server] ----------------
[server] header: Map()
[server] trace: ac746630-5636-4905-9f07-f9d88a1d2b7a
[server] request: id(5)
[server] result: \/-(Elem((tag,List((type,int))),List(Elem((text,List((content,5))),List()))))
[server] duration: 20253373 nanoseconds
[client] ----------------
[client] header: Map()
[client] trace: ac746630-5636-4905-9f07-f9d88a1d2b7a
[client] request: id(5)
[client] result: -\/(java.lang.IllegalArgumentException: 6 bits remaining: 0x00)
[client] duration: 231969252 nanoseconds
[error] (run-main-3) java.lang.IllegalArgumentException: 6 bits remaining: 0x00
java.lang.IllegalArgumentException: 6 bits remaining: 0x00
at scodec.interop.scalaz.package$AttemptSyntax$$anonfun$toTask$extension0$1.apply(package.scala:27)
at scodec.interop.scalaz.package$AttemptSyntax$$anonfun$toTask$extension0$1.apply(package.scala:27)
at scodec.interop.scalaz.package$AttemptSyntax$$anonfun$toTask$extension1$1.apply(package.scala:31)
at scodec.interop.scalaz.package$AttemptSyntax$$anonfun$toTask$extension1$1.apply(package.scala:31)
at scodec.Attempt$Failure.fold(Attempt.scala:113)
at scodec.interop.scalaz.package$AttemptSyntax$.toTask$extension1(package.scala:31)
at scodec.interop.scalaz.package$AttemptSyntax$.toTask$extension0(package.scala:27)
at remotely.package$$anonfun$evaluate$1$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(package.scala:76)
at remotely.package$$anonfun$evaluate$1$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(package.scala:70)
at scalaz.concurrent.Task$$anonfun$flatMap$1$$anonfun$1.apply(Task.scala:36)
at scalaz.concurrent.Task$$anonfun$flatMap$1$$anonfun$1.apply(Task.scala:36)
at scalaz.concurrent.Task$.Try(Task.scala:389)
at scalaz.concurrent.Task$$anonfun$flatMap$1.apply(Task.scala:36)
at scalaz.concurrent.Task$$anonfun$flatMap$1.apply(Task.scala:34)
at scala.Function1$$anonfun$andThen$1.apply(Function1.scala:52)
at scalaz.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:58)
at scalaz.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:58)
at scalaz.concurrent.Future.step(Future.scala:109)
at scalaz.concurrent.Future.listen(Future.scala:75)
at scalaz.concurrent.Future$$anonfun$listen$1$$anonfun$apply$4.apply(Future.scala:79)
at scalaz.concurrent.Future$$anonfun$listen$1$$anonfun$apply$4.apply(Future.scala:79)
at scalaz.Free$$anonfun$map$1.apply(Free.scala:52)
at scalaz.Free$$anonfun$map$1.apply(Free.scala:52)
at scalaz.Free.scalaz$Free$$fastFlatMap(Free.scala:71)
at scalaz.Free$$anonfun$resume$1.apply(Free.scala:87)
at scalaz.Free$$anonfun$resume$1.apply(Free.scala:87)
at scalaz.std.FunctionInstances$$anon$1$$anonfun$map$1.apply(Function.scala:56)
at scalaz.Free$$anonfun$run$1.apply(Free.scala:186)
at scalaz.Free$$anonfun$run$1.apply(Free.scala:186)
at scalaz.Free.go2$1(Free.scala:133)
at scalaz.Free.go(Free.scala:136)
at scalaz.Free.run(Free.scala:186)
at scalaz.concurrent.Future$$anonfun$async$1$$anonfun$apply$14.apply(Future.scala:376)
at scalaz.concurrent.Future$$anonfun$async$1$$anonfun$apply$14.apply(Future.scala:376)
at scalaz.stream.async.mutable.Queue$$anonfun$1.apply(Queue.scala:222)
at scalaz.stream.async.mutable.Queue$$anonfun$1.apply(Queue.scala:221)
at scalaz.stream.async.mutable.Queue$$anonfun$scalaz$stream$async$mutable$Queue$$stop$1$4$$anonfun$apply$8.apply$mcV$sp(Queue.scala:282)
at scalaz.stream.async.mutable.Queue$$anonfun$scalaz$stream$async$mutable$Queue$$stop$1$4$$anonfun$apply$8.apply(Queue.scala:282)
at scalaz.stream.async.mutable.Queue$$anonfun$scalaz$stream$async$mutable$Queue$$stop$1$4$$anonfun$apply$8.apply(Queue.scala:282)
at scalaz.concurrent.StrategysLow$$anon$3$$anon$4.call(Strategy.scala:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 2 s, completed 21.11.2016 22:11:05
So, I ran into a few issues with the macro stuff. First, the doc needs to mention that projects need to add the paradise plugin, otherwise the macro will fail silently and the eventual reference to .environment
on the server impl will cause a compilation failure.
addCompilerPlugin("org.scalamacros" % "paradise" % "2.0.1" cross CrossVersion.full)
Next, I was unable to get the macro working if I factored out the protocol. So the following works:
@GenServer(remotely.Protocol
.empty
.codec[String]
.codec[scalaz.\/[String, Int]] // fqn red'd even though it's imported
.specify1[String, scalaz.\/[String, Int]]("pop"))
abstract class World2Server
but if I try to factor out the protocol thus:
package demo
object WorldProto {
val definition = remotely.Protocol
.empty
.codec[String]
.codec[scalaz.\/[String, Int]]
.specify1[String, scalaz.\/[String, Int]]("pop")
}
@GenServer(demo.WorldProto.definition)
abstract class World2Server
compilation fails with the following message:
/Users/rnorris/Scala/doobie-remotely/src/main/scala/demo/World2.scala:26: exception during macro expansion:
scala.tools.reflect.ToolBoxError: reflective compilation has failed:
object definition is not a member of package demo.WorldProto
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.throwIfErrors(ToolBoxFactory.scala:315)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.compile(ToolBoxFactory.scala:249)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl.compile(ToolBoxFactory.scala:416)
at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl.eval(ToolBoxFactory.scala:419)
at scala.reflect.macros.runtime.Evals$class.eval(Evals.scala:16)
at scala.reflect.macros.runtime.Context.eval(Context.scala:6)
at remotely.GenServer$.impl(GenServer.scala:57)
@GenServer(demo.WorldProto.definition)
^
I tried putting the protocol in another compilation unit and even in another project, but it didn't make a difference. I avoid macros so I really don't know what I should expect here; hopefully it's something dumb.
The latest version of scalaz-stream
contains an implementation of uncons
that is generalized to any Monad
. It also contains a bug fix over the implementation found within the Remotely code base.
Implement a 0mq transport that uses the REQ
& REP
socket types. The intention here is that by making use of the already well-defined 0mq wire framing protocol, we can layer on application protocol versioning with relative ease. This also makes the integration path for non-scala services much smoother as they simply need to reuse an existing 0mq client in their language and use the primitives provided in a way thats compatible with the service.
Currently, it is possible to create any kind of Remote. This is not too much of an issue because remotes should be generated by the macro most of the time. It might still be interesting to associate the creation of a Remote with it's environment and/or protocol in order to make manual creation more typesafe. It would also in theory make the macro implementation more fool proof (in theory).
It might also allow to more easily assert the fact that it is only possible to stream one stream per Remote call (if we decide to go down that route).
RemoteSpec
relies on the order of ScalaCheck property evaluation in order to clean up the pool in the end. Here's what happens after upgrading to ScalaCheck 1.13.x:
[info] + Remote.roundtrip[List[Int]]: OK, passed 100 tests.
[info] Elapsed time: 0.630 sec
[info] + Remote.check-serializers: OK, proved property.
[info] Elapsed time: 0.000 sec
[info] + Remote.add3: OK, passed 100 tests.
[info] Elapsed time: 0.214 sec
[info] + Remote.roundtrip: OK, passed 100 tests.
[info] Elapsed time: 0.307 sec
[info] + Remote.check-declarations: OK, proved property.
[info] Elapsed time: 0.000 sec
[info] + Remote.cleanup: OK, proved property.
[info] Elapsed time: 0.002 sec
[info] ! Remote.roundtrip[Double]: Exception raised on property evaluation.
[info] > ARG_0: List()
[info] > ARG_1: Map()
[info] > Exception: java.lang.IllegalStateException: Pool not open
[info] org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
[info] org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
[info] org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
[info] remotely.transport.netty.NettyTransport$$anonfun$1.apply(Client.scala:36)
[info] remotely.transport.netty.NettyTransport$$anonfun$1.apply(Client.scala:35)
(more stack trace ...)
This blocks an upgrade to ScalaCheck 1.13.x. It can be easily fixed by converting it into a ScalaTest suite. See larsrh/remotely@30b0ae6 for an implementation.
helpful stuff from @ahjohannessen on gitter;
Should resetAllAttrs not be replaced by resetLocalAttrs in GenServer and GenClient -- this is recommended by http://docs.scala-lang.org/overviews/macros/changelog211.html#how-to-make-your-210x-macros-work-in-2110
I am trying to build remotely with Scala 2.11.5 here - I get things to compile after replacing resetAllAttrs, though there are some deprecation warnings.
Any reason for being on 2.10.4?
Btw, here is an example of using macros with both 2.10.4 and 2.11.5: https://github.com/scalamacros/sbt-example/blob/master/project/Build.scala
Signatures.wrapResponse
used by GenServer.scala returns invalid 'type parameter string' for list of parameters. There are missing parentheses around function parameters.
There is an error in SignaturesSpec.scala:
Signature("foo", List(Field("baz", "Baz"), Field("qux", "Qux")), "Zod").wrapResponse should be ("Baz,Qux => Response[Zod]")
should be:
Signature("foo", List(Field("baz", "Baz"), Field("qux", "Qux")), "Zod").wrapResponse should be ("(Baz,Qux) => Response[Zod]")
This causes error for protocols that use specify2.
Following protocol definition:
val definition = Protocol.empty
.codec[Int]
.specify2("f", Field.strict[Int]("a"), Field.strict[Int]("b"), Type[Int])
causes error during macro expansion:
[error] <toolbox>:1: wrong number of type parameters for method ref: [A](s: String)(implicit evidence$1: reflect.runtime.universe.TypeTag[A])remotely.Remote[A]
[error] val f = Remote.ref[Int,Int => Int]("f")
I can send two files to place in 'test' and 'test-server' modules that cause compile error seen above.
val definition = Protocol.empty.codec[String].specify1("type", Field.strict[String]("foo"), Type[String])
@GenClient(definition)
object FooClient
this throws compile error at
https://github.com/oncue/remotely/blob/master/core/src/main/scala/GenClient.scala#L46
since type
is a reserved word.
easiest solution could be using backtick to surround the variable name, like this,
val signatures : Set[Tree] = s.signatures.map { sig =>
c.parse(s"""val `${sig.name}` = Remote.ref[${sig.typeString}]("${sig.name}")""")
}
0mq 4 provides Curve based security; assess this and investigate if wrapping / proxying the connection via PKCS-based TLS would be in any way advantageous over Curve, or instead using 0mq 3.
// project 1
val definition =
Protocol
.empty
.codec[String]
.codec[Option[String]]
.codec[List[String]]
.codec[Map[String, String]]
.specify3("foo", Field.strict[String]("a"), Field.strict[String]("b"), Field.strict[String]("c"), Type[List[String]])
// project 2 depends on project 1
// I did use the full fqn of definition
@GenServer(definition) abstract class Foo
It throws exception,
[error] scala.reflect.macros.ParseException: ';' expected but ',' found.
[error] at scala.reflect.macros.contexts.Parsers$$anonfun$parse$1.apply(Parsers.scala:18)
[error] at scala.reflect.macros.contexts.Parsers$$anonfun$parse$1.apply(Parsers.scala:17)
[error] at scala.collection.mutable.LinkedHashSet.foreach(LinkedHashSet.scala:91)
[error] at scala.reflect.macros.contexts.Parsers$class.parse(Parsers.scala:17)
[error] at scala.reflect.macros.contexts.Context.parse(Context.scala:6)
[error] at scala.reflect.macros.contexts.Context.parse(Context.scala:6)
[error] at remotely.GenServer$.parseType(GenServer.scala:35)
[error] at remotely.GenServer$$anonfun$2.apply(GenServer.scala:59)
[error] at remotely.GenServer$$anonfun$2.apply(GenServer.scala:58)
[error] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
[error] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
[error] at scala.collection.immutable.Set$Set2.foreach(Set.scala:111)
[error] at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
[error] at scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
[error] at scala.collection.SetLike$class.map(SetLike.scala:92)
[error] at scala.collection.AbstractSet.map(Set.scala:47)
[error] at remotely.GenServer$.impl(GenServer.scala:58)
[error] @GenServer(....definition
I tried a few specify1
, and they are all good though.
Also, in three years the current ones expire, at which time someone will spend an afternoon figuring out why the tests are suddenly failing (been there done that).
I think addition of hlists to Codecs
has been more pain than anything else, compile times are staggering once your protocol grows. Honestly I do not see what value it brings about. This is the commit that introduced hlists on Codecs
:
@timperrett @stew @runarorama WDYT?
If reverting that change is is not desirable, could we then at least find a better way that does not make scalac crap out once the protocol starts to grow?
It's a copy paste from another test and does not test what it is supposed to based from the description
@ahjohannessen made a request in gitter that I think makes sense, which is that we could perhaps treat Environments as a monoid, so that they can be added together, and we can do something like:
(env1 |+| env2).serve(addr) ...
This would allow for remote functions to be logically grouped, and the later composed.
It's out of date
Probably using HList
s to encode what codecs are part of a given Environment
at the typelevel.
During development and debugging it is nearly always needed to do some ad-hoc requests to a given system. Given remotely is using strongly typed contracts, we must devise a way of dynamically generating a client-side understanding of the server shapes, and be able to construct a set of input devices based on those shapes.
@tpolecat points out that NettyTransport.single(addr) is side-effecty and should probably return a Task. I tend to agree, but it might gross up the client code a bit. Open the issue for discussion before I go making this change.
Could you please point me to an example repo using remotely?
Thanks
We originally chose our netty version to align with akka, but we can perhaps have both netty 3 and netty 4 on the classpath without conflicting because they use different package names.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.