unfiltered/unfiltered

A toolkit for servicing HTTP requests in Scala

Unfiltered

See the Unfiltered documentation for instructions on using the project.

Modules

library

The core application library for Unfiltered. This module provides interfaces and implementations of core request extractors and response combinators.

filter

Binds the core library to filters in the servlet API.

filter-async

Provides asynchronous support for the filter module

jetty

Provides an embedded web server abstraction for serving filters.

netty

Binds the core library to a Netty channel handler and provides an embedded server.

netty-uploads

Provides extractors for multipart posts using netty.

specs2

Provides helpers for testing Intents with specs2.

uploads

Provides extractors for multipart posts using the servlet API.

json4s

Provides extractors for working with jsonp and transforming json request bodies.

websockets

A minimal server websocket interface build on netty

Community

Join the Unfiltered mailing list on Google Groups.

Example Apps

There are some giter8 templates for Unfiltered contain a bit of example code.

• unfiltered-netty.g8 g8 template for netty webservers
• unfiltered-war.g8 g8 template configured with sbt war plugin
• unfiltered-websockets.g8 g8 template for websocket based chat app
• unfiltered.g8 basic g8 template example

unfiltered's Issues

deprecated keystore methods in jetty connector

[warn] /home/nathan/Programming/unfiltered/jetty/src/main/scala/secured.scala:40: method setKeystore in class SslSocketConnector is deprecated: see corresponding Javadoc for more information.
[warn]     setKeystore(keyStore)
[warn]     ^
[warn] /home/nathan/Programming/unfiltered/jetty/src/main/scala/secured.scala:41: method setKeyPassword in class SslSocketConnector is deprecated: see corresponding Javadoc for more information.
[warn]     setKeyPassword(keyStorePassword)
[warn]     ^
[warn] /home/nathan/Programming/unfiltered/jetty/src/main/scala/secured.scala:56: method setTruststore in class SslSocketConnector is deprecated: see corresponding Javadoc for more information.
[warn]   sslConn.setTruststore(trustStore)
[warn]           ^
[warn] /home/nathan/Programming/unfiltered/jetty/src/main/scala/secured.scala:57: method setTrustPassword in class SslSocketConnector is deprecated: see corresponding Javadoc for more information.
[warn]   sslConn.setTrustPassword(trustStorePassword)
[warn]           ^

adding multiple filters on same context path to jetty server only calls the last filter in chain

failing test cases are in PlanSpec

only one netty plan can run

The definitions for both netty channel and cycle plans respond with NotFound if the intent is not defined at the input. The net effect is that only one plan will ever be considered for handling (the last one added).

Docs for working with cookies

requested on the list

OAuth 2 support

why do you need OAuth 2 when there's OAuth already?

OAuth 2 is simpler because it doesn't try to reinvent https. The spec is still at draft, but I think it's worth jumping the gun. See Google and Facebook for their impl.

code?

Since my OAuth provider is at somewhere else, I've only implemented protection in my oauth2 branch. It's not as complete as OAuth support, but by writing the protection code first, it guarantees that the protection doesn't rely on the provider implementation.

The only thing the writer needs to provide is an implementation for AuthSource:

/** Represents the authorization source that issued the access token. */
trait AuthSource {
  def authenticateToken[T](token: AuthenticationMethod, request: HttpRequest[T]): Either[String, UserLike]

  def realm: Option[String] = None
}

AuthenticationMethod

OAuth2 supports different kinds of access tokens.

• bearer token type just includes the access token in the header or query param:

  GET /resource/1 HTTP/1.1
  Host: example.com
  Authorization: Bearer h480djs93hd8

• MAC token type includes the signature similar to OAuth 1:

  GET /resource/1 HTTP/1.1
  Host: example.com
  Authorization: MAC token="h480djs93hd8",
  timestamp="137131200",
  nonce="dj83hs9s",
  signature="YTVjyNSujYs1WsDurFnvFi4JK6o="

ProtectionLike vs Protection

I wanted it to have sensible defaults, but I also wanted it to be able to override it with my perverted needs (like returning error messages with 200 and jsonp when ok is truthy). Here's what I did:

1. Put implementation in trait ProtectionLike and let case class Protection extend from it.
2. Refactor out any params and responses to method, so I can override them later.

note

Like I said, I only have protection code, so I am not married to the design. It could evolve as the provider gets implemented or spec changes.

sample server

def setup = { server =>
  val source = new AuthSource {
    def authenticateToken[T](access_token: AuthenticationMethod, request: HttpRequest[T]): Either[String, UserLike] =
      access_token match {
        case BearerAuth("good_token")          => Right(new User("test_user"))
        case MacAuth("good_token", _, _, _, _) => Right(new User("test_user"))
        case _ => Left("bad token")
      }

    override def realm: Option[String] = Some("Mock Source")
  }

  server.filter(Protection(source))
  .filter(unfiltered.filter.Planify {
    case User(user) => ResponseString(user.id)
  })
}

fileuploads stream interface should return a user defined type

Unit 1 is weak. We should just use a generic type on the method

Invalid Header Parsing

We shouldn't expect all headers are comma an semicolon delimited.

Take http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.25 for example. Unfiltered current treats Sat, 29 Oct 1994 19:43:31 GMT as a Seq of 2 values Seq(Sat, 29 Oct 1994 19:43:31 GMT)

We should probably define a set of Header parsers instead of just using one for all headers.

default impl for netty channelHandler#exceptionCaught?

I've been bitten a few times now by not having implemented this [1]. Should we leave it up to the application developer to implement this? In some cases like planified intents they wouldn't be able to access it outside the partial function handler

Should we provide a default?

[1]:http://docs.jboss.org/netty/3.1/api/org/jboss/netty/channel/SimpleChannelHandler.html#exceptionCaught(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ExceptionEvent)

Scalate module missing from unfiltered-all in build.scala

Hi,

I just noticed the scalate module is missing in the "unfiltered-all" aggregate in build.scala.

Cheers,
Andrew

ensure dispatch http clients are propertly shutdown after tests

I see warnings from dispatch in tests about not being shutdown. We can prob add a convenience "loaner" style method in the Hosted trait which manages the shutdown itself.

static resource handling pamflet example

I think we could use an example of serving up static content using the jetty module, possibly under Bindings+and+Servers.html. I can take this.

Support for FilterOutputStreams, response refactor

It's difficult to use FilterOutputStream subclasses like GzipOutputStream in combination with Unfiltered's built-in response functions. It should be possible to prepend one of the to the response function chain and then write out a filtered response with another response function that is unaware of the filter.

Also, look at the scalate response function. It should be using a stream or writer so that it can benefit from this.

socket-io module

Seems to be the emerging standard for supporting browsers that don't support native WebSockets through a WebSocket-like interface. The spec may be a place to start. Also see the wiki. Found a java bridge based on netty. Totally doable

OAuth module depends on Authorization extractor

The Authorization(...) returns None in case the header is missing, so the whole thing doesn't match if the header is missing.

This should gracefully fallback on a query string based request

netty http server fails to handle multiple ChannelHandlers

Adding multiple ChannelHandlers through the netty http server's handler interface will result in an on the first request IllegalArgumentException: Duplicate handler name

Request extractors lose async subtype

Not a huge deal, but it means you have to write

case req @ GET(_) =>

instead of

case GET(req) =>

or else you can't call req.respond

Unfiltered-scalatest fetches old version of scalatest to classpath and throws up exception when using with scala 2.9.x

Here's an example project on how to reproduce the binary compatibility problem. Dependency should be declared as provided or something so it doesn't end up for the user of the library..

https://github.com/vvilhonen/binarybackward

signed cookies as sessions

A generic session feature can be implemented using just cookies. It would be cool to implement this using signed cookies.

unfiltered.netty.cycle.Plan evaluates Intent on IO worker thread

It is kind of uncool that unfiltered.netty.cycle.Plan executes on a worker thread, meaning it can only be used for request handlers that are known to be fast. People should use the channel.Plan for anything slow, but the problem with that is it doesn't cater to standard request cycles.

For typical request cycles it's probably better for us to maintain a thread pool executor, even though that requires making assumptions that I would rather leave up to the application. As long as users can override these assumptions (bring their own executor), it's a better default than leaving them to block an IO worker.

The trick is getting this to work in a way that is compatible with unfiltered.Cycle.Intent, which is the basis for the GZip kit and potentially many others.

Netty Http static resource handler

We can currently serve static resources with the jetty interfaces Http(...).resources(url) but not with the netty http interface. I'll add a method that supports the same feature using the same interface.

netty file uploads should not return keep-alive connection header

after some browser testing I noticed that a netty RecievedMessage#respond call will set a keep-alive connection header of the client requested so. This is fine except for 304 1 where the semantics are that the client already has an up to date copy of the resource requested. The spec also says to omit entity headers like content-length. The net effect for static resources in the browser is that the browser doesn't know when to close the connection so it remains open and blocks loading other resources on the page. We should just close the connection after sending the 304

Http listening interface

The jetty and netty server's share a very similar design for binding and listening on a port but share no common explicit interface/trait. Perhaps we can come up with one. Not sure if it will benefit us that much but it may be nice to have when designing consistent future backend server binding interfaces.

unfiltered-uploads seems to return an empty file on MultiPartParams.Disk(r)

Which is surprising from a pattern matches perspective, Seq(emptyFile) should be Seq.empty

prefer secure over isSecure

I must have missed this in the de-javaifcation refactor

https://github.com/unfiltered/unfiltered/blob/master/library/src/main/scala/request/HttpRequest.scala#L28

let's deprecate this and prefer request.secure instead

need a better way to handle exceptions in websockets plans

an exception can still go by quietly closing the connection

HostPort extractor should not return the port in the host part of the tuple

It would make more sense for this to return (hostnamewithoutport, port). At the moment it just returns (hostnamewithport, port). I think this is fixed in one of my other branches but I can always merge that later.

https://github.com/unfiltered/unfiltered/blob/master/library/src/main/scala/request/headers.scala#L127-136

common interface across servers

Currently the jetty and netty servers do not share an interface, because I never thought that people would want to use them generically. This was incorrect!

websockets intent needs a way to Pass

Because of the problem with big partial functions and for other reasons, we need a way for the websockets intent function to pass on a request even if it isDefined for that request, like the way you can return Pass for Cycle.Intent.

netty file uploads

Somewhere buried in here 1 are nestled clues for handling file uploads in netty in a style framed for netty. Review this and extract out something generic idiomatic in unfiltered. The resulting interface should be as close to unfilered-uploads 2 as possible. It may result in the renaming of the uploads module to servlet or filter-uploads to make a clear distinction between it an a netty-uploads module.

On a side note, this may tie in nicely with some of the async stuff @n8han is working on if clients wish optionally to post files in an async fashion.

Add Support for Matrix Parameters

It would be nice to add support for parsing URI matrix parameters. JAX-RS currently supports this via the @MatrixParam annotation. It is used to inject matrix parameters that have been parsed out of the URL.

For example, consider some REST resources that allows a client to search for sites within a certain number of miles of a given latitude and longitude looking for people that work at the sites and meet some minimum age criteria. The @MatrixParam provides an easy way to extract lat, lon, withinMiles, and minAge from these URLs. The first example provides access to sites while the second provides access to employees.

http://example.com/sites;lat=30.25;lon=-81.1;withinMiles=10
http://example.com/sites;lat=30.25;lon=-81.1;withinMiles=10/employees;minAge=20

As Nathan suggested on the mailing list this might be relatively easy to achieve in Unfiltered using a regex or parser combinator.

Exception handling

In the Netty module we have several needs for exception handlers:

• exceptions in netty IO threads, normally handled in an exceptionCaught override
• exceptions in the executor IO thread that we start for deferred execution

The first can be handled fine in a user-defined plan class or trait, but with the Planify factory there is no way for apps to define a handler. There should be a consistent way to do it across unfiltered; an implicit parameter?

params does not pick up multipart/form-data

Steps

1. Request POST

    <form action="http://localhost:8080/foo" enctype="multipart/form-data" method="POST">
     <input type='file' name='argf'/>
     <input type='submit' />
    </form>
   

2. Try grabbing it as params("argf")

Problem

It returns Nil

Expectation

param("argf") returns a seq of string representation of the passed in param.

It would also be nice to have some mechanism to grab the byte array for the param as multipart("argf"). I am only interested as String, but someone may want to upload binary file like mp3 or jpeg.

Project directory structure should follow package naming

Eclipse fails to attach the source code from unfiltered project source jars. The reason is that project source code structure doesn't match package names. e.g. class unfiltered.response.ResponseString is stored to file
response/writers.scala
more appropriate directory would be unfiltered/response.

Eclipse bug:
http://scala-ide-portfolio.assembla.com/spaces/scala-ide/tickets/1000042--source-not-found--error

This is definitely not a bug in Unfiltered but rather an attempt to enforce some best Scala and Java practices.

In long run I'd expect all major Scala tools to resolve this kind of issues automagically but in mean time perhaps we can stick to some old conventions to allow existing tools to cope, perhaps?

netty defrerals and executor exception handling

We may be able to capture and handle at least one runtime exception throw when submitting a callable to the deferrable executor 1. This can potentially throw a RejectedExceptionException 2 if the server is shutting down. I think I may have captured this case the the following gist

https://gist.github.com/02267eac2f32970f9cf2

I think the server was shutting down in this case which caused the rejected exception 3 It may be safe to catch that and check if the executor is shutdown or terminated before re-throwing the exception

Support HTTP sessions

I'd like to use Unfiltered like a "traditional" web framework and automatic session support would be very useful.

As a workaround I'm doing http.current.setSessionHandler(new SessionHandler) when starting the application and req.underlying.getSession in each request. While it works it looks like a workaround more than a feature and some more explicit support would be nice.

HttpOnly cookie support

Cookies by default are subject to xss vulnerabilities without an HttpOnly flag. This should be easy to add with netty and the servlet sec support. This will block the ability for in page javascript to steal your a domains cookies

possibly migrating to a pure scala websocket impl

Our current websocket implementation is based on what is currently available in netty 1 as a wrapper. With the upcoming websocket spec 2 right around the corner. It may be a good idea to start looking at it an maybe implementing our own encoder/decoders based on the updated spec.

Some things to think about though are browser compatibility. Currently our version works with chrome and the latest firefox (with the websocket setting enabled). If we upgrade to the latest spec before browsers to we won't get much benefit out of being standards compliant :)

This issue is more of a research and todo for the future than an immediate need so I'm jotting it down here.

scalate tests don't pass in scala 2.9

noticed this before remove 2.7.7 support

Netty plans do not support chunked requests

Exceptions fly when requests are chunked, as they often are when posting non-trivial amounts of data. This affects cycle and async plans.

If the chunk aggregator is low overhead we should just add it to the default pipeline. If it's something that people might reasonably not want to use we should make it a required mixin, like exception handlers.

Deprecate Optional

Our new min supported version of scala removes the need for it

Netty binding does not URL-decode query parameter keys

Whilst the netty binding correctly url-decodes query parameter values, it does not do the same for keys.

For example, a query string of the form "?param%3A1=value1" should be extracted to
"param:1" -> List("value1")
but is instead extracted to
"param%3A1" -> List("value1")

See: https://github.com/n8han/Unfiltered/blob/master/netty/src/main/scala/bindings.scala#L166

websocket module seems to be broke in the latest version of chrome

using unfiltered-websockets.g8 as a test case, I am unable to establish a websocket connection in the latest version of chrome.

I noticed chrome is not longer sending an Origin header with the ws request. Which does cause an exception but even with a fallback on a * origin, the protocol fails.

Chrome recently upgraded 1 to the latest (and final) version of the spec 2 which should resolve questionable security issues

As a sanity check I double tested using the tried and true html5 demo 3 app for websockets which also fails in chrome

Both examples do however work in when enabling the craftily hidden settings 4 in firefox further indicating that we just need to adjust for the updated version of the protocol.

Async example

Would be cool to have an example (g8 template?) that uses dispatch nio and responds asynchronously to a request.

Configurable output charset for ResponseWriter

Right now it is using the JVM default, which is not be a bad option for configuring it, but some people are going to want a direct interface.

May also need to think through the consequences of having removed the Writer from the response binding. It's redundant from the binding's perspective, but for servlet-filter bindings it would have used the charset configured in the servlet container.

Rename beany methods in HttpResponse

We have a squeaky clean request interface now but I just noticed our response interface still follows getter/setter naming conventions 1. Refactor these into concise names

migrate towards specs2

Eventually specs1 will no longer be maintained. Now may be a good time to start migrating towards specs to. Things to be mindful of, before/after hooks are still usefully for starting and stopping webservers.

work towards parallel test execution

we should be able to work around this 1 by using the server anylocal and maybe exposing the port info for the companion dispatch client instances

websockets http request request fails to pass

a websocket plan which fails to match on an http request should pass to the next upstream channel handler

Example using kit, making in-app kit

Show how to abstract out request+response functionality.

Example of a generic cycle plan

Documentation should show off the awesome powers of type variance!