http4s/http4s

A minimal, idiomatic Scala interface for HTTP

Http4s

Http4s is a minimal, idiomatic Scala interface for HTTP services. Http4s is Scala's answer to Ruby's Rack, Python's WSGI, Haskell's WAI, and Java's Servlets.

val http = HttpRoutes.of {
  case GET -> Root / "hello" =>
    Ok("Hello, better world.")
}

Learn more at http4s.org.

If you run into any difficulties please enable partial unification in your build.sbt (not needed for Scala 2.13 and beyond, because Scala 2.13.0+ has partial unification switched on by default)

scalacOptions ++= Seq("-Ypartial-unification")

Requirements

Running the blaze backend requires a modern, supported version of the JVM to build and run, as it relies on server APIs unavailable before JDK8u252. Any JDK newer than JDK8u252, including 9+ is supported.

Code of Conduct

http4s is proud to be a Typelevel project. We are committed to providing a friendly, safe and welcoming environment for all, and ask that the community adhere to the Scala Code of Conduct.

License

This software is licensed under the Apache 2 license, quoted below.

Copyright 2013-2021 http4s [https://http4s.org]

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

[http://www.apache.org/licenses/LICENSE-2.0]

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Acknowledgments

Special thanks to YourKit for supporting this project's ongoing performance tuning efforts with licenses to their excellent product.

http4s's Issues

Bug when parsing URI with a fragment

When calling Uri.fromString a fragment will be ignored.

"parse absolute URI with parameters and fragment" in {
  val u = Uri.fromString("http://foo.bar/foo?bar=baz#Example-Fragment").get
  u should equal(Uri(Some("http".ci), Some(Authority(host = RegName("foo.bar".ci))), "/foo", Some("bar=baz"), Some("Example-Fragment")))
}

"parse relative URI with fragment" in {
  val u = Uri.fromString("/foo/bar#Examples_of_Fragment").get
  u should equal(Uri(path = "/foo/bar", fragment = Some("Examples_of_Fragment")))
}

Rename HTTP methods shorthands to be more natural to the specification

We should think about to hold the shorthands of HTTP methods in upper case in our DSL. This feels more natural to me.

package object dsl {
  val OPTIONS = Method.Options
  val GET = Method.Get
  val HEAD = Method.Head
  val POST = Method.Post
  val PUT = Method.Put
  val DELETE = Method.Delete
  val TRACE = Method.Trace
  val CONNECT = Method.Connect
  val PATCH = Method.Patch
}

What do you think?

Drop joda-time dependency

• The Scala community offers no standard date-time solution.
• joda-time will be forgotten in favor of JSR 310.
• JSR 310 is not available until Java 8, which we otherwise don't mandate.
• threetenbp works on all supported JVMs, but not in the same namespace as JSR 310.
• We can't wait for years until the JVM world has united on JSR 310.

Decide how to handle error in situations concerning invalid input

This is important to things like EntityDecoder. Two obvious solutions are:

• Use an either type such as the scalaz \/ or, less desirable, the stdlib Either
• Make a coherent Exception structure and leverage the exception capabilities of scalaz Task

I personally think the second could be more elegant, if we can pull it off, but I think that has been attempted and failed numerous times throughout history.

Blaze backend defaults to chunked-encoding for HTTP/1.1 requests regardless of Content-Length header

Per RFC 2616, the response must not have a Content-Length header and a non-identity transfer encoding (section 4.4).

WSFrame trait is closed

I wanna build a Process[Task, WSFrame] but the trait org.http4s.websocket.WSFrame is closed and custom cases (for example JSON) cannot be implemented straight away. I would suggest to use a Type Class Pattern here to decouple the details how to write a WSFrame. What do you think?

Fill out the tests

As we leave the realm of 'exploring the model' and enter the realm of some semblance of stability, we need to implement _lots_ of tests.

Provide Akka support

While Akka goes against everything this library represents, mainly composability and type safety, it is still a useful tool because of its simplicity, performance, and development velocity.

I first inclination is that it should be a http4s module, much like the core-dsl, so that it remains current yet optional.

Check binary compatibility between bugfix releases

We are starting to get enough interest that people are submitting bug issues/patches which is really awesome. In order to get these out there fast and ensure it doesn't ruin anyones day, we need to start checking for binary compatibility.

I don't have much experience with this, but allegedly this goal can be checked by adding the mima plugin to the sbt build project.

Provide uniform interface for backends to implements timeouts

Vaguely related to issue #9
We need to provide timeout support at the backend level, and it would be nice if configuring the timeout could be done uniformly.
On the topic, I suspect there are times where an exception may be necessary such as when switching to the websocket protocol when you may want to change or disable the timeout.

Set IP address via BlazeServer.newBuilder

It would be nice to be able to set an IP address via BlazeServer.newBuilder and not only the port. Does it make sense?

Deemphasize "service"

A user with similar goals was put off because of the emphasis on the word "service." HttpService is only in the server project, and has nothing to do with the coming client. We're good at services, but need to make more clear the "s" is Scala, not services.

Blaze server never responds if a Task[Response] is a failure

Minimum failing example:

    val service: HttpService = {
      case GET -> Root  => Task.fail(new Exception("zomg"))
    }

    BlazeServer.newBuilder
      .mountService(service, "/")
      .run()

Note that if an exception is thrown synchronously when generating the response, everything works fine.

Need a template for getting started

I'm willing to entertain arguments why it should be activator. But there should be something to get people up and running quickly.

Remove Config dependency

We can provide parameters and sensible defaults. Let applications configure themselves to their own standards. This is an area that http4s should remain unopinionated.

Blaze client never sends query string in requests

val uri = Uri.uri("http://localhost:8080/my/path?queryParam=value")
SimpleHttp1Client.decode[String](Request(uri = uri))(_ => EntityDecoder.text)

If you run a simple socket listener (nc -l 8080) and then execute the above, you'll see that the HTTP request does not include the query string:

GET /my/path HTTP/1.1
Host: localhost:8080
Content-Length: 0

Looking at Http1ClientStage.encodeRequestLine, the problem seems pretty obvious-- it's just using only the path property of the URI:

req.method ~ ' ' ~ uri.path ~ ' ' ~ req.httpVersion ~ '\r' ~ '\n'

This makes the client basically unusable for my purposes.

Avoid Using Reflection (including Manifest/ClassTag), use Macros Instead

This is what I suppose is a general jibe of mine, but I am not the biggest fan of reflection (either in the weak form, i.e. ClassTag/Manifest, or in the strong form, which is TypeTag)

Historically, reflection has been a failure. It brings many compatibility issues (import compiler/reflection and binary dependencies). It has also been broken (reflection used to not work under multithreading conditions, which has forced people like @casualjim to create his own reflection library/factory to deal with this). Even in its currently 'fixed' form, performance is degraded in multithreaded environments due to it using locks. Its also incredibly hard to debug, we have had issues in with reflection breaking code in production due to the reasons above.

Using reflection also introduces the risk of adding incompabilities if people want to build/use their own libraries ontop of http4s that use reflection. I personally have zero issues if people want to use reflection in their own custom stack above http4s, but using reflection in the base http4s core (which is designed to be as performant, safe and stable as possible) in my opinion isn't desirable, mainly due to past experiences of using Scalatra (which is also a minimalist framework) including reflection in its core libraries

What are peoples opinions of forcing (as part of the http4s style code) of not using scala reflection in http4s due to the reasons above? Currently there are only a couple of places where there is reflection (HeaderTag and AttributeMap) which I am willing to replace with macros, but if we all agree with this limitation sooner rather than later, I think it would be beneficial for http4s long term goal (and it can even be a selling point of http4s, considering the pain that reflection has caused with the scala community in general)

Finish the move to specs2

http4s uses typelevel stuff pretty much everywhere so specs2 seems like a good idea. Many modules are already moved.

Avoid usage of HttpServletResponse.setStatus(int, String)

We should avoid the usage of HttpServletResponse.setStatus(int, String) in Http4sServlet.

Maybe we can follow the suggestion: Deprecated. As of version 2.1, due to ambiguous meaning of the message parameter. To set a status code use setStatus(int), to send an error with a description use sendError(int, String). Sets the status code and message for this response.

Document new Service / HttpService functions

Website currently refers to a PartialFunction[Request, Response]. We just improved it to a Service. The site needs to catch up.

Add Logging middleware for http client and server

Ideally I would like to be able to log all http requests and response

Handle async timeouts gracefully in servlet

A async servlet timeout should kill the response task instead of leaving the response task to throw a curious exception after the rug has been pulled out from under it.

Setting headers within EntityResponseGenerator does not work

When writing

Ok("Some text", Headers(`Content-Type`(MediaType.`text/plain`)))

the response contains still application/octet-stream in the response.

Travis does not like pull requests

See #27 for an example. It's trying to parse the secrets when it need not.

Create an HTTP client

Reuse the existing request response model to implement a client.

Support Servlet 3.1 async reads and writes

Should be a progressive enhancement, not required. We still want to run on Servlet 3.0.

whitespace in param

hey! having some trouble with the router in http4s v0.4.0.

object X extends ParamMatcher("x")
object Y extends ParamMatcher("y")

class App(hostname: String = "localhost", port: Int = 9999)(implicit ec: ExecutionContext = ExecutionContext.global) extends Logging {
  def run = BlazeBuilder.bindHttp(port, hostname).mountService(service, "/").run.awaitShutdown()
  def logreq(r: Request): Unit = log.info(s"${r.remoteAddr.getOrElse("null")} -> ${r.method}: ${r.uri.path} ${r.uri.query}")

  lazy val service = HttpService {
    case req @ GET -> Root / "blah" :? X(x0) +& Y(y0) =>
      logreq(req)
      log.info(s"x0=${x0}, y0=${y0}")
      Ok("x = " + x0 + "\n" + "y = " + y0)
    case req =>
      logreq(req)
      NotFound("404: not found")
  }
}

(here's the full file: https://gist.github.com/sshastry/874fa6ce2431e3725e9f)

http://localhost:8080/blah?x=abc&y=ijk works and binds "abc" to x0 and "ijk" to y0.

but http://localhost:8080/blah?x=a+bc&y=ijk falls through to the NotFound route, contrary to my expectation that it would bind "a bc" to x0. likewise for http://localhost:8080/blah?x=a%20bc&y=ijk.

this is what the log shows:

14:47:43.370 [pool-6-thread-1] INFO  org.app.App - 127.0.0.1 -> GET: /blah Some(x=a bc&y=ijk)

something simple I'm overlooking? thanks!

Make up our minds on the signature of HttpService

Do we want to go with OptionT[Task, Response] or Task[Option[Response]]? I personally like the latter for simplicity or reading, but I'm not so opinionated as to keep holding this train up.
We should take a sort of 'vote' where we rank our interest in the two contenders from 0 to 10 with 10 being 'greatest interest'

My Vote
OptionT[Task, Response]: 4
Task[Option[Response]]: 6

If this is a stupid idea, close the issue, but I would like to get back to coding instead of bikeshedding. I find myself wandering across other languages (I discovered that D is pretty awsome if you need low level, and I found myself preferring Haskell to Rust which was completely unexpected) and its a waste of time.

EntityDecoder.collectBinary dies if the response body is empty

If there is no content, calling this method throws an exception:

java.lang.UnsupportedOperationException: empty.reduceLeft
    at scala.collection.TraversableOnce$class.reduceLeft(TraversableOnce.scala:165)
    at scala.collection.AbstractTraversable.reduceLeft(Traversable.scala:104)
    at scala.collection.TraversableOnce$class.reduce(TraversableOnce.scala:193)
    at scala.collection.AbstractTraversable.reduce(Traversable.scala:104)
    at org.http4s.EntityDecoder$$anonfun$collectBinary$1.apply(EntityDecoder.scala:76)

URI translation may be broken

run example, load address: http://localhost:8080/foo

13:55:21.064 [pool-1-thread-1] ERROR org.http4s.blaze.pipeline.Stage - Error running route: Request(GET,/foo,HTTP/1.1,Headers(Host: localhost:8080, Connection: keep-alive, Cache-Control: max-age=0, Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8, User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36, Accept-Encoding: gzip,deflate,sdch, Accept-Language: en-US,en;q=0.8),Halt(End),org.http4s.AttributeMap@12328f00)
examples java.lang.StringIndexOutOfBoundsException: String index out of range: -3
examples at java.lang.String.substring(String.java:1875) ~[na:1.7.0_71]
examples at org.http4s.server.middleware.URITranslation$.org$http4s$server$middleware$URITranslation$$transReq$1(URITranslation.scala:18) ~[classes/:na]
examples at org.http4s.server.middleware.URITranslation$$anonfun$translateRoot$1.apply(URITranslation.scala:21) ~[classes/:na]
examples at org.http4s.server.middleware.URITranslation$$anonfun$translateRoot$1.apply(URITranslation.scala:21) ~[classes/:na]
examples at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47) ~[na:na]
examples at org.http4s.server.Service$$anonfun$orElse$extension$1.apply(Service.scala:57) ~[classes/:na]
examples at org.http4s.server.Service$$anonfun$orElse$extension$1.apply(Service.scala:57) ~[classes/:na]
examples at org.http4s.server.Service$.apply$extension(Service.scala:21) ~[classes/:na]
examples at org.http4s.server.blaze.Http1ServerStage$$anonfun$runRequest$1.apply(Http1ServerStage.scala:120) ~[classes/:na]

Figure out body parsers

Writable is looking a lot better after today. Need a uniform idiom for reading bodies.

Figure out how to embed actual code examples in docs

I eyeballed it for the 0.2 release. That's not a good way to run a project, especially as the docs mature.

API version is scattered throughout the build

Search and destroy references to "0.2" in .travis.yml and *.sbt, or we're going to miss some when we jump to 0.3.

Response uses wrong header key for setting cookies

Uses "Cookie" instead of "Set-Cookie". Pull request to follow.

Make a more compelling example

ExampleService is a hodge-podge as we iterate on the model, but doesn't provide a real compelling welcome to a new user checking us out. Some of the routes are illustratitve (e.g., /ping, /echo), some are not (e.g., /challenge). We should split the documentation from the science experiments.

Deemphasize the status reason on Status

The reason-phase of a HTTP response is falling out of style and should be considered more of a guideline than a rule. RFC 7230 section 3.1.2 suggests that the reason-phrase should be ignored by the clients. The current http 2.0 draft doesn't even have the concept of a reason-phrase.

Because of these considerations, I think we should remove the fromIntAndReason method from the Status object and move the reason field to the second constructor parameter list. This will make equality checks safer.

Replace scalalogging with log4s

We use scalalogging-2.x for cross build support. A user is now trying to use scalalogging-3.x and hitting binary compatibility issues. We can't upgrade because scalalogging-3.x dropped Scala 2.10 support. That's the second time churn in this library has caused us pain. Let's use log4s/log4s.

HttpBin.org clone

I like the routes of http://httpbin.org. Might make an interesting example to clone it.

Provide hooks for instrumentation of server workings

Acceptance: we should be able to integrate metrics without a core dependency on metrics.

(Edit: we've decided an http4s-server dependency on Metrics is acceptable.)

Use MIMEDB to autogenerate MIME types

There is a library called MIME-DB (https://github.com/jshttp/mime-db) which is basically a JSON index of every known MIME type according to two sources, that also identifies if the types are compressible, and their known extensions

Would be ideal if we could integrate this as part of the build process (download the .json, parse it, and generate a Scala source file similar to what is happening in https://github.com/http4s/http4s/blob/master/core/src/main/scala/org/http4s/MediaType.scala#L213-L377)

Note that, the index is currently missing whether a MIME-Type is binary or not, however this is an open issue, so we should only migrate when they implement this jshttp/mime-db#3

NumericParam and inheriting classes should be renamed

We can write cases to match a Request with a path variable very easy, but the name NumericParam and their inheriting classes can be misunderstood and should be renamed to PathVar because a parameter is always in the Query String.

  val service: HttpService = {
    case Get -> Root / "hello" / LongParam(id) => Ok(s"id: $id")
    case r => NotFound("404 Not Found: " + r.pathInfo)
  }

Bug when parsing short IPv6 address

It seems I found a bug when parsing a short IPv6 address.

    "parse short IPv6 address" in {
      val s = "01ab::32ba:32ba"
      Uri.fromString("01ab::32ba:32ba").get should equal(Uri(authority = Some(Authority(host = "01ab::32ba:32ba"))))
    }

More concise way to bind services to servers

I am not a fan of the "intent" and "plan" lingo, but I like the way Unfiltered services are deployed: http://unfiltered.databinder.net/Bindings+and+Servers.html. Look to that for inspiration.

Make building a HTTP request less of a pain

Right now requests have to be built in the form

val req = Request(requestUri = Uri.fromString("/hello").get, body = body)
                  .withHeaders(Headers(Header.`Content-Length`("foo".length)))

which is unacceptable for generating requests for future client support.

Use JAWN for parsing bodies into Json ASTS

https://github.com/non/jawn

Why?

Jawn accepts parsing form various forms, including a ByteStream, which makes sense, since we HTTP4s works with byte streams. This can help reduce memory usage, and also possibly increase performance? Jawn appears to require a java.nio.ByteStream type to parse from a ByteStream
Parser is pure scala
Also available for 2.10.x
Competitive with parsers like jackson and GSON
It supports various backends that are external from the compiler, full list is

• argonaut
• json4s
• play
• rojoma
• rojoma-v3
• spray

Provide a type class to add custom param to an Uri

here is how I would like to use it:

case class Ttl(value: FiniteDuration)

val uri: Uri = ...
uri.withParam(Ttl(2.seconds))

Atm, I created a very simple type class to do the job:

trait QueryParam[T]{
  def withParams(params: IList[T])(uri: Uri): Uri
  def withParam(param: T)(uri: Uri): Uri = addParams(IList(param))(uri)
}

implicit class QueryParamOps(uri: Uri){
  def withParams[T: QueryParam](params: IList[T]): Uri =
    QueryParam[T].addParams(params)(uri)

  def withParam[T: QueryParam](param: T): Uri =
    QueryParam[T].addParam(param)(uri)
}

If you like the idea I can make a PR

Separate http4s Macros into their own project

After a lengthy discussion with @rossabaker, the best way to prevent any leaking of scala-reflect is to separate macros into their own project. Normally this wouldn't be needed, but the usage of QuasiQuotes plus scala 2.10.x support means that we need to pull in scala-reflect to compile the macros for scala 2.10.x.

If we separate out any usages of macros into their own sub project, make that subproject have a scala-reflect as a provided dependency, then the projects that depend on that sub project (in this case, http4s), won't pull the scala-reflect dependency.

Im fairly sure this is how it works, as I have done it before, will attempt this during the weekend, or early next week

Using ParamMatcher when matching Request

I want to describe a parameter when matching Request but the following does not compile. Maybe I missed something?

  object Limit extends IntParamMatcher("limit")
  val service: HttpService = {
    case Get -> Root / "hello" :? Limit(l) => Ok(s"limit: $l")
    case r => NotFound("404 Not Found: " + r.pathInfo)
  }

From Uri to String

We should enhance the Uri implementation to convert back to a String.

Uri
  .fromString("https://github.com/http4s/http4s/commits/develop?page=2")
  .toString must be equalTo
    "https://github.com/http4s/http4s/commits/develop?page=2"

Than I can reuse this case class in my own data structures to increase safeness by working with validated URIs. What do you think?

Give WebSocket some love

WebSockets and HTTP/2.0 will become increasingly important. In the mid future we should give some serious consideration to how they fit into our world.