vijayeluri/pipes-1

Pipes feed your threads

Pipes

Pipes is an easy way to parallelize long running pieces of code that produce objects that other long running pieces of code will use as input.

For example, think about a web crawler, a parsing logic and a service layer. What you do at the beginning is a repetition of these three steps: crawl, parse, save, for each URL you have to crawl. You soon understand you are wasting time: you are using just one core of your quad-core server. These steps can run independently: the crawler will crawl and feed the parsing logic ASAP; the parsing logic, as new web pages can be parsed, will produce objects for the service layer.

What's in the name

                     in-pipe          out-pipe   in-pipe          out-pipe
>> INCOMING DATA >>  ======= [ PUMP ] ======== - ======= | PUMP | ======== >> OUTGOING DATA >>
                                           pipes link

The building block of Pipes is the pipe: each pipe has a pump in the middle. The pump draws data from the incoming part of the pipe, elaborating it, to the outgoing part. Each middle pipe has its outgoing part connected with the incoming part of the next pipe.

You can assemble the pipes on your own, or ask a plumber to do that for you: you need to give the plumber the beginning and ending parts of your pipes, and the collection of pumps that will elaborate/transform the data. The plumber will assemble the pipes and the pumps.