Comments (13)
I managed to make a small prototype:
module Main where
import Control.Monad.IO.Class
import qualified Data.Conduit as C
import Data.Conduit.Internal
import qualified Data.Conduit.List as CL
import Data.Void
main :: IO ()
main = do
let printTee x = liftIO $ putStrLn $ "tee " ++ show x
let printNormal x = liftIO $ putStrLn $ "normal " ++ show x
let ping = liftIO $ putStrLn "ping"
putStrLn "\n -- Test 1 --"
CL.sourceList [1..3::Int] C.=$= tee (CL.mapM_ printTee)
C.$$ (CL.mapM_ printNormal)
putStrLn "\n -- Test 2 --"
CL.sourceList [1..3::Int] C.=$= tee (C.await >>= printTee)
C.$$ (CL.mapM_ printNormal)
putStrLn "\n -- Test 3 --"
CL.sourceList [1..3::Int] C.=$= tee ((C.await >>= printTee) >> ping)
C.$$ (CL.mapM_ printNormal)
putStrLn "\n -- Test 4 --"
CL.sourceList [1..3::Int] C.=$= tee ((C.await >>= printTee) >> ping >> (C.await >>= printTee))
C.$$ (CL.mapM_ printNormal)
-- | Forks a stream, like the Unix `tee` utility.
-- Like GNU tee, it yields each value first to the outer pipe, then to the inner one.
tee :: Monad m => Sink a m () -> ConduitM a a m ()
tee (ConduitM sStart) = ConduitM $ go sStart
where
go s = case s of
HaveOutput _nextPipeCannotHappen mFinalizer void -> absurd void -- TODO where run finalizer?
NeedInput f _fEnd -> NeedInput (\i -> yield i >> go (f i)) (\() -> Done ())
Done () -> idP -- The tee pipe stopped accepting; we don't care.
-- The main pipe then just passes values through.
PipeM mPipe -> PipeM $ do pip <- mPipe
return $ go pip
Leftover pip x -> yield x >> go pip -- TODO Is it right that we insert the x here?
It has a simpler type signature than the tee
in Data.Conduit.Extra.Pipes
, and doesn't have the lift
problems I mentioned (it composes).
Can you see any problem with this?
from conduit.
I think the leftover handling is incorrect; that clause should probably be:
Leftover pip x -> Leftover (go pip) x
though I haven't thought about it too much. I'd also probably prefer running the Sink to completion just in case it has any finalization it needs to perform. Other than that, seems good.
from conduit.
Looks like a much better tee
than what I wrote.
from conduit.
@jwiegley Do you want to pull this code into conduit-extra then?
from conduit.
@snoyberg Will do.
from conduit.
@snoyberg Where did it end up?
from conduit.
I did not get around to moving this into conduit-extra, so if that's still needed let me know. I can move that in today actually if it hasn't been already.
from conduit.
@jwiegley I don't need it right now, but I've found it very useful in the past, and would appreciate it to be somewhere upstream rather than in my own code.
Also what @snoyberg said above would have to be addressed.
from conduit.
OK, one issue I see: If NeedInput is resolved and passes a value down to the sink and downstream, and the sink then uses leftover
, downstream will see the leftover value again.
@snoyberg @nh2 Here is my version based on the comments above:
-- | Forks a stream, like the Unix `tee` utility.
--
-- This like GNU tee, but yields each value first to the outer pipe, then to
-- the inner one.
tee :: Monad m => Sink a m () -> ConduitM a a m ()
tee (ConduitM sStart) = ConduitM $ go sStart
where
go s = case s of
-- The sink passed to tee cannot generate values.
HaveOutput _ _ o -> absurd o
-- Whenever a value is needed by the sink, we ask upstream and then
-- yield the value to both the sink and downstream. Note that if
-- upstream terminates early, we pass the result value to the sink.
NeedInput f lc -> NeedInput (\i -> yield i >> go (f i)) (go . lc)
-- Once the sink is Done, switch to being a transparent pass-through.
Done () -> idP
-- If an action must be executed to determined the next @Pipe@, do so
-- and then process it with this function.
PipeM mPipe -> PipeM $ liftM go mPipe
-- A leftover from the sink will be passed again to both the sink and
-- downstream, meaning downstream will see the same result twice even
-- though it never used 'leftover' itself.
Leftover pip x -> Leftover (go pip) x
from conduit.
I don't like the behavior of Leftover
. I think it would make more sense to add a call to injectLeftovers
around sStart
.
from conduit.
@snoyberg Yes, that sounds nicer. Here is the result:
-- | Forks a stream, like the Unix `tee` utility.
--
-- This is like GNU tee, but yields each value first to the outer pipe, then
-- to the inner one.
tee :: Monad m => Sink a m () -> ConduitM a a m ()
tee (ConduitM sStart) = ConduitM $ go (injectLeftovers sStart)
where
go s = case s of
-- The sink passed to tee cannot generate values.
HaveOutput _ _ o -> absurd o
-- Whenever a value is needed by the sink, we ask upstream and then
-- yield the value to both the sink and downstream. Note that if
-- upstream terminates early, we pass the result value to the sink.
NeedInput f lc -> NeedInput (\i -> yield i >> go (f i)) (go . lc)
-- Once the sink is Done, switch to being a transparent pass-through.
Done () -> idP
-- If an action must be executed to determined the next @Pipe@, do so
-- and then process it with this function.
PipeM mPipe -> PipeM $ liftM go mPipe
-- There can be no leftovers from the sink, since we called
-- 'injectLeftovers' above.
Leftover _ x -> absurd x
from conduit.
Actually, this now looks like just a special case of passthroughSink
. The latter function provides the ability for the Sink
to return arbitrary return values and have them consumed by some finalizer function.
from conduit.
Ok, then I guess we leave this as closed and defer to passthroughSink
. I was unaware of that function.
from conduit.
Related Issues (20)
- `ResourceT` is subject to the `forever` space leak
- slidingWindowC makes consuming tail elements inconvenient HOT 3
- Quadratic performance on left-associated binds HOT 6
- resourcet needs unliftio-core >=0.1.1.0 lower bound
- "stdout" test stalls temporarily sometimes HOT 3
- Implementing delay operator HOT 3
- Clarification regarding `sinkFile` and `sinkFileBS` HOT 1
- Compatibility with mtl-2.3 HOT 5
- Lots of inline-rule-shadowing warnings
- Accumulate ByteStrings into single result HOT 3
- bracketP with custom monad stack HOT 1
- Flaky test suite failure with GHC 9.4.2 / text-2.0.1 HOT 6
- *> leaks space where >> doesn't - so should the *> implementation be optimized? HOT 1
- Confusion with `sourceProcessWithStreams` and `takeExactly` HOT 1
- `sourceProcessWithStreams` without hijacking `stderr`? HOT 2
- Support incrementally feeding into Conduits HOT 1
- Better performance with a left associative `.|`? HOT 2
- Functor instance causes memory leak when used with Shake
- Running a conduit in a function with an unrelated parameter twice leaks memory
- network-conduit-tls-1.4.0 test suite build failure with crypton-connection-0.4 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from conduit.