justsml/escape-from-callback-mountain

Example Project & Guide for mastering Promises in Node/JavaScript. Feat. proposed 'Functional River' pattern

Escape from Callback Mountain v2.6.0

Refactoring JavaScript w/ Functional River Pattern

I am a big fan of Functional Programming and Modular JavaScript. This project's goal is to demonstrate the latest Functional Promise patterns, while taking you through a refactor of real world callback-based NodeJS/JavaScript.

What is the Functional River pattern?

It is an async & sync version of the Collection Pipeline pattern.

Your parameters/data represents the water, and functions form the riverbed.

---

Roughly speaking, my definition of pipeline is a sequential series of chained functions where arguments line up with return values (using Array methods, Promises, or similar). Key to my adaptaion is using named functions.

This ultimately results in your code reading like a step-by-step story.

---

Compare these 2 examples:

// ❌ Non-Functional River / Collection Pipeline Code ❌
const formatScores = scores => scores
  .map(x => x * 2.0)
  .map(x = x.toFixed(2))

// ✅ Functional River Code ✅
const formatScores = scores => scores
  .map(double)
  .map(formatNumber)

Let's look at a more complex example, with asynchronous requirements added in the mix...

Comparison: Callbacks vs. Functional River

See both Before and After examples below.

Before

Node-style Callbacks w/ Nesting

Note: This is intentionally reasonable callback code. Even if nested. Not trying a straw-man attack.

After

'Functional River' Pattern

The technique I demonstrate hopefully illustrates the Functional River pattern:

Functional River Goals/Benefits:

• Higher level logic implemented with multiple smaller single-purpose functions, assembled to read like a story.
• Decoupled modules are easier to maintain & upgrade over time.
• Reduce bugs by relocating ad hoc logic. (e.g. one-off transformations, untested validation)
• Use same interface for both synchronous & asynchronous code. (promise.then(value => alert(value)))
• Prefer immutable, stateless code as essential building blocks.
• Less elaborate, modular code is naturally more reusable.
• Easier to move logic around - rebundle simple functions as needed to create new higher-order functions.
• Increased testability - eliminate hidden surface area.
• Substantially faster code readability - versus artisinal functions assembled with ad hoc code glue (a Big Ball of Mud).

Note: The Functional River Relies on ideas from Lisp to SmallTalk - adapted to a JavaScript world. Apologies to Promise Resistance Leader Brian Leroux. For alternative patterns please read my more detailed article demonstrating 4 JavaScript Composition Techniques (with Examples)

Have feedback, fixes or questions? Please create issues or Pull Requests. Or DM me at twitter @justsml.

If you feel this subject has already been exhauted, please see my post Beating a dead horse?

Key Steps

1. Step 1: Break Up The Big Functions - read the code: PR #2: Flatten Functions
2. Step 2: DRYer Code - read the code: PR #3: DRYer Code
3. Step 3: Cleanup Code - read the code: PR #5: Post Cleanup

Pros & Cons

Pros

• Less ad hoc code results in:
  • More uniform code between different teams & developers,
  • Performance tooling & refactoring is an appreciably better experience,
  • More certainty about code correctness,
  • Higher code reuse.
• 100% Unit Testability
  • Unit tests uniquely prove you found, understand, AND resolved a given bug,
  • Faster bug resolution process,
• Flatter code hierarchy == less filler to remember
  • Re-organizing code is easier & less prone to bugs - with Pure-ish Functions

Cons

• Performance. I've run some micro-benchmarks - it's not awesome. However, 3 important things:
  1. It's not meaningfully slower in real world applications.
  2. If it is necessary, performance analysis & tuning is a much improved experience. Smaller functions make it easier to see where slow code lurks - especially if you profile unit tests.
  3. As more people adopt these patterns, things will improve. V8/Chrome has been impressively fast at optimizing for emerging patterns.
• Debugging can be more difficult. Though I have updated my dev tricks to debug this style of code, even without the confort of Bluebird's error handling. I'll add more sample scripts for this later.
• Something new to learn. Deal with it, you're a developer.
• If you have an existing project with lots of code, the unfortunate reality is: Refactors Suck.
• EventEmitter- & Stream-based code is not improved much, if at all, using this technique. Look into RxJS
  • Ongoing experiments include simple closures, extend Promise with EventEmitter, or using Bluebird's .bind to inject variable state into the Promise chain. (I know, "ugh side-effects, gross." PRs welcome.)

Pattern Showdown

View Callback Comparison

Summary

It's perhaps true that an overly-done flat & modular JS Project can feel more disorganized over time. New practices & approaches must be explored (from monorepos, to breaking up modules when-needed to meet org/dev/deployment needs).

Project and code discipline is just as important as it's always been. Also, the community is still developing consensus around Functional JS patterns, immutability and overall project organization.

When done right, one of Functional River's greatest strengths is the ability to relocate & rearrange modules with low risk. If this still feels risky, your modules are probably still too entangled (coupled).

---

Ultimately my goal is to better understand & advance Modular + Functional JS patterns. Hopefully I can interest some of the skeptics along the way 🤞

---

Please Star this project ❤️

---

Wiki Contents

Wiki Main

• Steps
  1. Step 1: Break Up The Big Functions
  2. Step 2: DRYer Code
  3. Step 3: Cleanup Code
• Related Articles
  • Finding Promise Anti-patterns
  • What about Async, Await and Generators?
• Examples
  • HTTP Client Work Queue
  • Typed Errors
  • S3 File API w/ Express Middleware & Class wrapper

---

Credits & Inspiration

I highly recommend reading (or watching) every single link here.

• Tom Stuart's Purely Amazing Programming with Nothing
• Eric Elliot's Tour de force: The Two Pillars of JavaScript
• Ashley Williams Speaks The Modular Tructh in A Brief History of Modularity | JSConf EU 2017
• And Eric's Rise and Fall and Rise of Functional Programming (Composing Software) is damn good too ;)
• Martin Fowler expertly lays down the Ruby + Haskell pipeline in Collection Pipeline

---

v2.0 Released

Escape from Callback Mountain Key Updates

1. README now more focused on the Functional River pattern.
   • Counter-examples are still included in ./src, just not featured on README.
2. There's an updated production-ready library Functional Promises which grew out of the feedback & research from this Project.

escape-from-callback-mountain's Issues

Graphics need to be consistant

Fix different annotations & background color on README's screenshots.

Add live coded video refactors

Add rich media demos!

Show concurrency limits & timeout example

To prevent prematurely creating Promises and exhausting resurces, we'll start our Promise chain by simply passing our parent function's params (ids array) into Bluebird .map handler calling our api method - now with sane limits:

// In order to avoid overwhelming `api.getById` - for example lets assume we are limited to 4 concurrent connections we can easily add or remove this kind of real-world requirement 
const getUsersById = (ids = []) => Promise.resolve(ids)
  .map(api.getById, {concurrency: 4})

// Call getUsers, but attach logic to extract names out
const userNames = getUsersById([1, 2, 3])
  .all().spread((users) => users.map(u => u.name))
userNames.then(console.log)) // > ['Emily', 'Mel', 'Paris']

// Or reuse Promise w/ built-in memoization. 
//     +1 for caching lookups + disposing of objects in RAM using JS scope & garbage collection. 
const getUserNamesAndActivity = (userIds) => {
  const users = getUsersById(userIds)
  const userActivity = users.map(({username}) => api.getTopActivity({username}), {concurrency: 4}) // throttle net-bound API calls
  const userNames = users.map(({name}) => name) // no I/O or net-bound call: don't throttle
  return Promise.props({userNames, userActivity});
}

This "requirement" has a funny way of coming up really late in the game, maybe even in production or perhaps after a massive spike in traffic.

Avoid issues caused by "launching" too many Promises at once: simply 'start' with an array of primitive values or with something like: Promise.resolve(Array.from({length: 5000}))
Then you just need to call .map and specify the concurrency limit, like so:

Promise.resolve(Array.from({length: 5000}))
  .map((x, idx) => api.activityByIndex(idx))

Thanks for the kind words in https://github.com/justsml/escape-from-callback-mountain/wiki/Beating-a-dead-horse%3F

If you have any feedback about how native promises can give users a better debugging experience please let us know :) We're gathering use cases for Node.js to see what we can do better.

Create example w/ Async/Await

Add screen/Animated GIF to docs:

Show example with bluebird collection methods

Show custom error example

Using filtering with Bluebird's .catch w/ 2 params like:

Promise.resolve()
  .catch(NetworkError, _fatalError)
  .catch(NetworkError, _fatalError)
...

Show how to handle sloppy var hoisting/side-effects

Demonstrate a few ways to use closures for sharing variables over boundaries.

Don't write code like this - ok if you cant re-order function calls, or a use a context-free wrapper function. I sometimes fallback to this technique if I have a legacy API, or other design challenge (e.g. EventEmitter with several paths for success & errors).

Create RxJS Example

Using methods from http://reactivex.io/documentation/operators/flatmap.html and

Add lib & test code

Add Folder Structure to README

Add info on folder structure, start small, add more complex stuff to the examples folder.
Or maybe do a wiki page on suggestions/thought process.