chrisdinn/brando

A Redis client written with Akka's IO package

brando's Issues

Migration guide from v2 to v3

Previous versions of brando implemented Akka Stash and queued requests when the redis was not connected. As part of #46 this behavior was silently dropped and version 3 throws an exception instead, which is significantly different behavior.

1. I suggest to document this change because I spent several hours debugging why the application is not working after the upgrade.
2. It would be wonderful to provide some kind of migration guide or and simple option to re-enable this behavior. I really miss it. In asynchronous connection, I don't really think it must be handled by the application itself if the connection can be (re-)establish within a certain timeout.

Thanks!

Classes need to be in a suitable package

In order to keep things clean, the package hierarchy needs to be set up. Something along the lines of com.digital_achiever.brando.

Add Event Messages for Connection Status

It's currently difficult to determine the state of connections in Brando.
Tcp.ConnectionClosed, Tcp.Connected etc should send events (possibly to the parent or to listeners passed in) so that services can respond accordingly.

Doc link is broken

The doc link in the README is broken

Authentication doesn't work

I traced issues reported as Brando #64 and play-redis #44 and #29

Backstory

We noticed the first occurrence of the issue on Heroku because it secures its redis instance. It caused excessive logging without any reasonable error message.

However, I successfully reproduced in on localhost when I secured my redis instance.

How to reproduce

Enable redis authentication in redis.conf, e.g., uncomment

requirepass foobared

Code reproducing the issue

Example of Play framework controller:

package controllers

import javax.inject.Inject

import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._

import play.api.mvc._

import akka.actor.ActorSystem
import akka.pattern._
import brando.Redis

class Application @Inject( )( system: ActorSystem ) extends Controller {

  implicit val timeout = akka.util.Timeout( 1.second )

  val actor = system actorOf Redis( "localhost", 6379, database = 1, auth = Some( "foobared" ) )

  def ping( ) = actor ? brando.Request( "PING" )

  def index = Action.async {

    ping( ).map { case message =>
      Ok( views.html.index( "done", message ) )
    }.recover { case ex =>
      Ok( views.html.index( "error", ex.toString ) )
    }
  }

}

Observation

When used Redis actor the response is the redis is disconnected. When the StashingRedis is used, requests timeout. Both no further reason provided.

Tracing the issue

To trace the issue, I inserted several logging statements into Brando. I figured out two things:

1. Authentication method is never called even when proper auth is provided. The reason is it stucks in Connecting state but the authentication is executed on Connected.
2. The following logging statements suggest that Brando always creates new connection with every attempt but probably somehow does not closes the previous. First, actor id changes, second, when deployed on Heroku, it dies after 30 attempts due to 'ERR max number of clients reached'.

2016-04-21T13:07:53.071807+00:00 app[web.1]: Queue(Actor[akka://application/temp/$v])
2016-04-21T13:07:54.081759+00:00 app[web.1]: connecting
2016-04-21T13:07:54.093458+00:00 app[web.1]: Some(Failure(brando.RedisException: NOAUTH Authentication required))
2016-04-21T13:07:54.093624+00:00 app[web.1]: Queue(Actor[akka://application/temp/$w])
2016-04-21T13:07:55.112304+00:00 app[web.1]: connecting
2016-04-21T13:07:55.119680+00:00 app[web.1]: Some(Failure(brando.RedisException: NOAUTH Authentication required))
2016-04-21T13:07:55.119837+00:00 app[web.1]: Queue(Actor[akka://application/temp/$x])
2016-04-21T13:07:56.135040+00:00 app[web.1]: connecting
2016-04-21T13:07:56.140825+00:00 app[web.1]: Some(Failure(brando.RedisException: NOAUTH Authentication required))
2016-04-21T13:07:56.141029+00:00 app[web.1]: Queue(Actor[akka://application/temp/$y])
2016-04-21T13:07:57.152416+00:00 app[web.1]: connecting
2016-04-21T13:07:57.161871+00:00 app[web.1]: Some(Failure(brando.RedisException: NOAUTH Authentication required))
2016-04-21T13:07:57.162046+00:00 app[web.1]: Queue(Actor[akka://application/temp/$z])
2016-04-21T13:07:58.184895+00:00 app[web.1]: connecting
2016-04-21T13:07:58.200840+00:00 app[web.1]: Some(Failure(brando.RedisException: NOAUTH Authentication required))
2016-04-21T13:07:58.210118+00:00 app[web.1]: Queue(Actor[akka://application/temp/$A])
2016-04-21T13:07:59.221666+00:00 app[web.1]: connecting
2016-04-21T13:07:59.226342+00:00 app[web.1]: Some(Failure(brando.RedisException: NOAUTH Authentication required))
2016-04-21T13:07:59.226689+00:00 app[web.1]: Queue(Actor[akka://application/temp/$B])
2016-04-21T13:08:00.243840+00:00 app[web.1]: connecting
2016-04-21T13:08:00.250774+00:00 app[web.1]: Some(Failure(brando.RedisException: NOAUTH Authentication required))
2016-04-21T13:08:00.251051+00:00 app[web.1]: Queue(Actor[akka://application/temp/$C])
2016-04-21T13:08:01.264740+00:00 app[web.1]: connecting
2016-04-21T13:08:01.273008+00:00 app[web.1]: Some(Failure(brando.RedisException: NOAUTH Authentication required))
2016-04-21T13:08:01.273249+00:00 app[web.1]: Queue(Actor[akka://application/temp/$D])
2016-04-21T13:08:02.293235+00:00 app[web.1]: connecting
2016-04-21T13:08:02.297606+00:00 app[web.1]: Some(Failure(brando.RedisException: ERR max number of clients reached))
2016-04-21T13:08:02.297748+00:00 app[web.1]: Queue(Actor[akka://application/temp/$E])

Subsequent issues

When max number of clients is reached, it crashes at this line.

Finally, it goes out of memory after while. However, I am not sure what exactly causes the memory leaking, just guessing.

HA issue - ShardManager will try to send a connect message to a dead actor if it receives an updated shard

It appears if brando has a shard connected to a dead redis instance that actor will die.
If we try to send a new shard to the shard manager for that host, then it will send a Connect message to the dead actor so it can never resolve in some failover scenarios.

I'll look at the code a bit more later - these are high level observations.

Course of action is to make the connections immutable and just poisonpill and recreate connections on Shard messages to the shard manager.

Cannot recover from `Connection refused` during connect/reconnect

When testing the recovering of disconnects by creating a ipfw rule on OSX, I find that Brando doesn't seems to recover.

To block/allow a port I'm using :
sudo ipfw add 1 deny tcp from any to any 6379
sudo ipfw del 1 deny tcp from any to any 6379

Seems like when the connection is setup, Brando is able to detect the disconnection when the rule is set. However, it cannot connect or reconnect.

It might be an akka IO issue I just posted something on the user list : https://groups.google.com/forum/#!topic/akka-user/sL_wL_OQgbo

Add Batch Support for Sharding

accept something like (key:String, batch: Batch).
Would be useful to have a way to do multi/exec, watch etc with sharding when doing multiple operations on a shard.

Strange Exception

Do you know what might generate this exception? I am finding this using play-redis, which has a dependency on brando, on Heroku.

a.a.OneForOneStrategy - queue empty java.util.NoSuchElementException: queue empty at scala.collection.mutable.Queue.dequeue(Queue.scala:66) ~[org.scala-lang.scala-library-2.11.7.jar:na] at brando.Connection$$anonfun$receive$1$$anonfun$applyOrElse$4.apply(Connection.scala:91) ~[com.digital-achiever.brando_2.11-3.0.3.jar:3.0.3] at brando.Connection$$anonfun$receive$1$$anonfun$applyOrElse$4.apply(Connection.scala:78) ~[com.digital-achiever.brando_2.11-3.0.3.jar:3.0.3] at brando.ReplyParser$class.parseReply(ReplyParser.scala:145) ~[com.digital-achiever.brando_2.11-3.0.3.jar:3.0.3] at brando.Connection.parseReply(Connection.scala:27) ~[com.digital-achiever.brando_2.11-3.0.3.jar:3.0.3] at brando.Connection$$anonfun$receive$1.applyOrElse(Connection.scala:78) ~[com.digital-achiever.brando_2.11-3.0.3.jar:3.0.3] at akka.actor.Actor$class.aroundReceive(Actor.scala:467) ~[com.typesafe.akka.akka-actor_2.11-2.3.13.jar:na] at brando.Connection.aroundReceive(Connection.scala:27) ~[com.digital-achiever.brando_2.11-3.0.3.jar:3.0.3] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [com.typesafe.akka.akka-actor_2.11-2.3.13.jar:na] at akka.actor.ActorCell.invoke(ActorCell.scala:487) [com.typesafe.akka.akka-actor_2.11-2.3.13.jar:na]

ReplyParser should cache parse progress on failure

ReplyParser re-parses the entire reply bytestring every time it gets new bytes from Redis. If you get lots of small responses from Redis (which happens, especially with commands like SMEMBERS) it can be surprisingly slow.

Repository Down

As I write this, it looks like your repository is down.

Brando is not in your releases

https://github.com/chrisdinn/chrisdinn.github.com/tree/master/releases

Batched requests can get sent out of order

Batched requests can get sent out of order when a given request fails to write to the tcp socket. The connection will get a Tcp.CommandFailed message and then retry the message. But at this point it is possible other requests have already been sent successfully.