Improving threading performance

[ucb-bar]

Improving threading performance

[schoeberl]

On 26 Aug, 2019, at 20:41, Richard Lin ***@***.***> wrote: ZIO? Akka? I've heard that Scala prefers Monads / etc for the problems that coroutines solve. Need to do more research into options to speed up the concurrency.

Akka (Scala) uses the Java/JVM threads internally. So the threading performance will not improve. However, maybe they use more lightweight synchronization than locks. I assume you use plain Java locks for synchronization. Cheers, Martin

[ducky64]

So today during chisel-dev, @jackkoenig (I think?) linked a page on fibers in Cats Effect. The page doesn't talk too much about API or implementation, but the short of it is that for server applications, you have a lot of concurrent tasks to serve requests (which we like to write as straight-line code) that are going to be frequently blocked (eg, on external IO), and simple approaches including one-thread-per-request or thread pools have significant performance issues. So the overall idea is "fibers" (suspendable tasks written as straight-line code) that could be remapped between threads as needed.

That being said, note that we don't actually want parallelism (concurrency in chiseltest is just a programming model, the threads are actually run sequentially and deterministically). But the suspendable-without-threads is useful.

Fiber API example

See the API examples in the Cats Effect tutorial. My takeaway from that is (and I might be wrong, this is making a lot of inferences from skimming that page):

• Fundamentally, Cats Effect / fibers do not solve the problem of coroutines without user-facing API changes. However, it does try to preserve the 'gist' of the straight-line code style while trying to minimize additional boilerplate (eg, compared to if you had to explicitly write a cooperatively-multitasked system).
• Here's a simple example of how Cats Effect does sequential actions, from their Fiber scaladoc

  val launchMissiles: IO[Unit] = IO.raiseError(new Exception("boom!"))
  val runToBunker = IO(println("To the bunker!!!"))
  
  for {
    fiber <- IO.shift *> launchMissiles.start
    _ <- runToBunker.handleErrorWith { error =>
      // Retreat failed, cancel launch (maybe we should
      // have retreated to our bunker before the launch?)
      fiber.cancel *> IO.raiseError(error)
    }
    aftermath <- fiber.join
  } yield {
    aftermath
  }

  Also consider checking out the examples in the tutorial.
• Ignoring the cancelability, the syntax is significantly different than straight line code. Also, note that all actions need to be wrapped in IO, and (not visible here) this is used to encapsulate (and make explicit) the state / context normally implicit in straight-line code (roughly a "Monad" I think).
• This is (I think?) what the Scala Couroutines tried to do (and hide details from the user) with a compiler plugin. It had many limitations and seems to be no longer maintained.

Implications for ChiselTest (or: my tl;dr)

• It does not appear Fibers would allow us to support improved threading performance without user-facing API changes.
• We can take inspiration (and potentially code?) from their suspendable-without-threads implementation. But this would require a significant user-facing API change, though this might be limited to writing library code.
• A "hybrid" option may be possible, where libraries can be written to either target the suspendable-without-threads API or the naive simple-but-threaded API. I haven't fully thought through what that would look like, but the idea is that most users would be able to use the current simple API, and if they encounter performance bottlenecks, they could gradually port code to the faster suspendable-without-threads (but more complex) API. Combined with scheduler / threading optimizations (eg, fast paths for threaded code), this might be a feasible compromise solution? (but there's also the possibility that it's "worst of both worlds"
• One thing to do might be to implement the fast-path optimizations and see how much that addresses problems. For example, if that solves everything, then great; we're done. If the bottleneck stems from threading overhead from use of a few library functions we control, then that supports implementing a Fiber-like fast API. Otherwise, if it's all in user concurrent libraries, and we deem the Fiber-like API too complex, then there's no good solution (yet?).
• I also heard Java coroutine support discussed during the meeting today. All this might be obsolete if that were to happen, and Scala were to pipe through support for that. But that's also a question of when.

[jackkoenig]

Yeah, you're right. Cats Effect and other libraries that provide an IO monad have ways of wrapping imperative snippets (not everything needs to be a for-comprehension), but the ability to preempt a Fiber relies on a more FP API of passing values rather than side-effecting statements. I was originally thinking we could just use the IO Monad underneath the user-API, letting them write lots of side effects, but then Cats Effect wouldn't be able to pre-empt the thread at cycle boundaries anyway. For something like this to be useful, you'd essentially need a flatMap between any two operations that impact the design (eg. any step or a poke based on the value in a peek). I guess the real difference between co-routines and functional fibers is side effects: co-routines let you write some sort of yield side-effecting statement while these fibers require input-to-output value propagation.

A functional effect-based API could work and has some desirable properties, but it would be an entirely different API.

Per the "hybrid" option, I'm not sure how well that would work. I think the benefits of Cats Effect and whatnot come from everything using the IO Monad, so long as something is using the non-FP API, I don't think we'd see any benefits. I could be wrong though.

[ducky64]

I think with the hybrid option, it just means the scheduler needs to handle both threads and fibers, where fibers wouldn't incur a thread switch. I'm more concerned about the mixed programming model and usability, but it's possible I've overlooked something with implementation difficulty.