pheymann/typedapi

Build your web API on the type level.

typedapi's Issues

Add server-side support

A first implementation could use a Trie built from all uri-method combinations defined and link to function calls on the leaves.

See #6 for implementation details.

Add support for akka-http

A first implementation could use Http.singleRequest which should be straight forward.

Improve exponential implicit search space

Currently, we experience an exponential growth of compile times with increasing API sizes. Taking a look at compiler statistics shows that we waste a lot of time during implicit resolution (as expected):

API:

:= :> Query[String]("test1") :> Query[String]("test4") :> Query[String]("test2") :> Query[String]("test3") :> Put[Json, User]

Statistics (-Ystatistics:typer):

time spent in implicits       : 276 spans, 1635ms (90.1%)
   successful in scope         : 129 spans, 1159ms (63.9%)
   failed in scope             : 147 spans, 1372ms (75.6%)
   successful of type          : 49 spans, 1554ms (85.6%)
   failed of type              : 98 spans, 1190ms (65.6%)
   assembling parts            : 144 spans, 59ms (3.3%)
   matchesPT                   : 11475 spans, 172ms (9.5%)
time spent in macroExpand     : 216 spans, 1351ms (74.4%)

The time to do the macro expansions significantly increases with the number of elements. My best guess why that is happening are the following actions:

• Lazy expansion during TypelevelFoldLeft derivation
• repeated Witness derivation (API type creation and during RequestDataBuilder/RouteExtractor derivation

We also face the problem of multiple passes of the API type. But that should "only" add a constant factor to the compile time.

An indicator that the recursive Lazy resolution during the TypelevelFoldLeft derivation could be the main problem is given by the fact that Witness derivation during API-to-type transformation needs considerable less time:

time spent in implicits       : 28 spans, 139ms (27.5%)
   successful in scope         : 0 spans, 0ms (0.0%)
   failed in scope             : 28 spans, 88ms (17.5%)
   successful of type          : 6 spans, 38ms (7.7%)
   failed of type              : 22 spans, 11ms (2.2%)
   assembling parts            : 28 spans, 3ms (0.6%)
   matchesPT                   : 430 spans, 10ms (2.0%)
time spent in macroExpand     : 5 spans, 24ms (4.9%)

To get a complete picture I need to profile the compiler. But before doing that I will start by implementing a low hanging fruit which is fusing the different derivation passes. Instead of having a fold, request data, etc. pass I will fuse them into a single pass. That way I should get rid of the recursive Lazy and spare me the repeated iterations.

If the problem still persists I have to invest more time and take a deeper look.

Support for header families

Very recently i came across a certain limitation of typed-api, as title suggests it is about header families, for example all headers starting with l5d-ctx (linkerd context). Program may not be interested to use all of the headers of that family but it might be required to forward those when doing request to some other service. Unfortunately it is impossible to do this given the current API, but i think it should be pretty simple to add something like the following:

  val MyApi = "header" :> "family" :> Header[String].family("my-hdr-family") :> Get[Json, A]

Given the following definition one can interpret it as to extract all headers starting with my-hdr-family on the server-side, and to provide zero or more headers on the client side.

Add alternative API definition syntax

Their should be an alternativ definition syntax for APIs for people not familiar with Servant. One way could be to make it look more like an HTTP route (as proposed in reddit):

val Api = api(
  method = Get[A], 
  path = "hello" / "world" / Segment[Int]('name), 
  query = Query[Int]('id) :: Query[String]('sortBy), 
  header = Header[String]('agent) :: Header ..., 
  body = ReqBody[User]
)

Add a Swagger support

It would be great if it will be possible to generate a swagger json file for the server side API.

ClientManager enhancment

Following the great effort made to publish 0.1.0 I tried to incorporate typedapi to my new project unfortunately i encountered the following limitation of the ClientManager. It forces user to specify port on which the following endpoint is available but unfortunately it is not covering all possible cases, as address without port does not necessarily mean that request will be made using protocol default port (443 or 80). I think that this issue can be fixed quite easily without breaking the existing code with the following implementation of ClientManager:

/** Provides a supported client instance and some basic configuration. */
final case class ClientManager[C](client: C, host: String, port: Option[Int]) {

  val base = port match {
    case Some(p) => s"$host:$p"
    case None => host
  }
}
object ClientManager {
   def apply[C](client: C, host: String, port: Int) = ClientManager(client, host, Some(port))
}

Header handling improvement

There are still limitations regarding headers. Let me elaborate:

Server.Match

Server can accept any header which name contains given string, but only its value is forwarded to the library user, which is quite frankly not very useful . Hopefully it should be quite easy to fix, i propose the following solution:

IsHeader typeclass

A typeclass could be introduced to indicate that a specific type may represent a header. So integration with specific servers would be even better.
Pseudo-signature

trait HeaderDecoder[H] {
  def parse(key: String, value: String): Either[Error, H]
}

Api definition

def myApi[H: HeaderDecoder] = := myPath :> Server.Match[H]("my-hdr")

Server usage

//usage with http4s
import org.http4s.Header
def implementation(headers: Set[Header]): = ???
derive[IO](myApi[Header]).from(implementation)

It would require only a small change to RouteExtractor.scala to work i think.

(Proposed) Client.Headers

This change was previously rejected and is not mandatory because it can be properly handled by reimplementing integration for specific http client, question is does it need to be done this way. I think the use-case is quite common, especially in combination with Server.Match: service accept a specific group of headers (used for request tracing etc.) and forwards them to different services.

Strange api transformation behaviour

Recently i encountered very strange bug, that was very hard to pinpoint. I will provide example (maybe not minimal but this exact combination causes the failure, in this case UUID as param type and Server.Match element) that should reproduce on version 0.2.0.
I have the following endpoint definition:

val Api = := :> Segment[UUID]("param") :> Server.Match[String]("headers") :> ReqBody[Json, TypeA] :> Put[Json, TypeB]

Now i work with it as always, so i define serialization/deserialization mechanism for request/response, have to define ValueExtractor for UUID and then derive endpoint from it and mount it (using http4s) as backend (i wrote custom integration with the newest version but it is exposing the same types as the standard one):

val endpoint = derive[F](Api).from(func)
val sm = ServerManager(BlazeServerBuilder[F], "0.0.0.0", 8080)
mount(sm, endpoint)

The following code does not compile resulting with compiler error like the following:
could not find implicit value for parameter executor: typedapi.server.EndpointExecutor.Aux[Req,typedapi.shared.SegmentInput :: typedapi.shared.ServerHeaderMatchInput :: shapeless.HNil,String("param") :: String("headers") :: Symbol with shapeless.tag.Tagged[String("body")] with shapeless.labelled.KeyTag[typedapi.dsl.MT.application/json.type,Symbol with shapeless.tag.Tagged[String("body")]] :: shapeless.HNil,java.util.UUID :: scala.collection.immutable.Map[String,String] :: TypeA :: shapeless.HNil,typedapi.shared.PutWithBodyCall,this.Out,F,TypeB,Resp]

I started digging into this error and what i found out is that somehow this.Out is infered to the wrong type by the compiler, or something like that because when i wrote the following custom http4s executor, the same endpoint compiled :

trait SimpleExecutor[R, VIn <: HList, Rout, F[_], FOut, Out] {
  def execute(req: R, eReq: EndpointRequest, endpoint: Endpoint[_, _, VIn, _, Rout, F, FOut]): Either[ExtractionError, Out]
}

//instance very similar to the default http4s executor instance
implicit def noReqBodySimple[VIn <: HList, ROut, F[_], FOut](implicit
    encoder: EntityEncoder[F, FOut],
    ME: Effect[F]
  ): SimpleExecutor[Request[F], VIn, ROut, F, FOut, F[Response[F]]] = ???

//with body instance
implicit def withReqBodySimple[VIn <: HList, ROut <: HList, Bd, F[_], FOut](implicit
    encoder: EntityEncoder[F, FOut],
    ME: Effect[F],
    decoder: EntityDecoder[F, Bd],
    _prepend: Prepend[ROut, Bd :: HNil]
  ): SimpleExecutor[Request[F], VIn, (BodyType[Bd], ROut), F, FOut, F[Response[F]]] = ???

//mounting function, again analogous
def mountHttp4s[VIn <: HList, ROut, M <: MethodType, F[_], FOut](
    server: ServerManager[BlazeServerBuilder[F]],
    endpoint: Endpoint[_, _, VIn, M, ROut, F, FOut]
  )(implicit
    executor: SimpleExecutor[Request[F], VIn, ROut, F, FOut, F[Response[F]]],
    mounting: MountEndpoints.Aux[BlazeServerBuilder[F], Request[F], F[Response[F]], Resource[F, Server[F]]]
  ): Resource[F, Server[F]] = ???

The most important thing i changed is that i erased the information that Rout is equal to VIn and simply casted the value when needed (assuming that it would be equal anyway which only made sense).
Now using mountHttp4s code compiles and runs but what is happening is that Rout is actually a reversal of VIn so i get the following error message when trying to invoke this endpoint:
scala.collection.immutable.Map$EmptyMap$ cannot be cast to java.util.UUID
I checked the original definition of executor for http4s and did not found any part which "reversed" input argument.

I am really puzzled here because the issue seems to be non-existent when using Get method (standard mount works great), it's not only request body issue because for example Delete also fails to compile (and fails at runtime using my special Executor implementation)

Get rid of `:= :>`

I had to use some initial empty state to guarantee that "path" is translated into a Witness. Why? Because an extension of String using implicit class was not able to derive a Witness, as Scala/Shapeless is not able to proof that the given String is a literal/singleton.

Another way could be to change the associativity (:>: instead of :>) and create GetCons, PutCons, ... as initial states with an empty HList type. Thus, we should be able to write:

val Api = "find" :>: Segment[String]('name) :>: Get[User]

Move type level fold completely to the type level

Right now we need a carrier object to store the type of the fold somewhere. But I think it isn't necessary. The classes would look something like this:

sealed trait TypeLevelFoldFunction[H, In] {
  type Out
}

object TypeLevelFoldFunction {
  ...

  def at[H, In, Out0]: Aux[H, In, Out] = new FoldFunctionHelper[H, In] {
    type Out = Out0
  }
}

sealed trait TypeLevelFoldLeft[H <: HList, Agg] {
  type Out
}

implicit def hnilCase[Agg]: TypeLevelFoldLeft.Aux[HNil, Agg, Agg] = new TypeLevelFoldLeft[HNil, Agg] {
  type Out = Agg
}

implicit def foldCase[H, T <: HList, Agg, FtOut, FOut](implicit f: TypeLevelFoldFunction.Aux[H, Agg, FtOut], 
                                                                next: Lazy[TypeLevelFoldLeft.Aux[T, FtOut, FOut]]): TypeLevelFoldLeft.Aux[H :: T, Agg, FOut] = new TypeLevelFoldLeft[H :: T, Agg] {
  type Out = FOut
}

Improve `implicit not found` exceptions

Right now these exceptions can look cryptic if a user is not familiar with the inner workings of the library. I already tried to improve the message but I think it isn't understandable enough yet. An example:

• MergeToEndpoint tries to find an RouteExecutor for every endpoint
• these executors try to find encoders/decoders (at least for http4s, but should be the some for every framework)
• if no encoders/decoders are in place Scala cannot find an executor and therefore is not able to find a merger instance. The user on the other side just gets the information that no merger could be found with a note that no executor was available.

Error handling

Motivation

At least for now there is no way of providing your own methods of handling errors, so any error in your service end up being reported as internal server error to the client. This can be addressed in a project by reimplementing integration with your http-framework of choice. But i think that providing facility to enable custom error handling is quite important. There are certain kinds of failures that are not internal server errors, or implementation issues like resource with given id missing, that should return proper status code and error information to the caller.

Proposal

Signature of Get[MT <: MediaType, A] and other methods respectively could be changed to something like Get[MT <: MediaType, E, A]. Then specific http-framework integrations would have to care about exposing facility to handle errors of type E in a specific way.

Add support for Finagle/Featherbed

Looks like a great library. I could suggest adding support for something like GitHub.com/finagle/finch (server) and GitHub.com/finagle/featherbed (client).