robinmalfait / lazy-collections Goto Github PK
View Code? Open in Web Editor NEWCollection of fast and lazy operations
License: MIT License
Collection of fast and lazy operations
License: MIT License
This package could provide utilities for a bit of "eager" processing of async iterators.
A "buffer" of a specified size to process items of an iterator in parallel before resolved items are yielded enables:
map
.race()
where first resolved items are yielded first.eager()
for guaranteed item order.This suggested style of functions could take one argument that describes the desired "parallelism" of the operation.
The argument could be a simple integer, or even better, a function that continuously supplies an adjusted "buffer" size to process in parallel before resolved items are yielded (good for backoff when an API is rate-limited, you probably want to switch down to a buffer size of 1
and then increase again slowly).
If race(5)
is in the pipe, immediately call next()
5 times on the iterator and put them into a processing "buffer" (probably a map). When any resolves, remove it from the buffer, adjust the buffer size, fill up the buffer again using next()
(unless the iterator reported being done
), then yield the resolved item.
eager(5)
would do the same thing, but keep resolved items until they can be yielded in the same order they arrived.
This is an idea I've had since I did a similar thing in a project for my former employer last year.
Had this package existed at that time it would have been a very elegant base for what I did!
I basically implemented map
, flatten
and tap
for async iterators in similar ways to this package (I think those were the only basic operations I needed) then I added the race-style-parallelism on top.
The use case was reading a lot of paginated data from the Google Classroom APIs, in multiple steps. We wanted a list of a student's assignments, with course and teacher data for each assignment, something like this:
For a typical student that could entail up to a 100 API calls, and if done sequentially, it took a loooong time...
By reading data from an async iterator in each step and "racing" 10-20 items from each of them, the next page of data was available just ahead of time, requests from all steps could be processed in parallel and it was all in nice readable chained collection operations. Ok, they would've been even more readable with the nice pipe-style API of this package ๐
Of course this wouldn't have been an issue at all if the Google Classroom API was GraphQL ๐คท and the API limits was a big thing as we really hammered it with short bursts!
Hi RobinMalfait,
I like your lib but it require node 13.x, can you make it work with node 12.x?
Regard,
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.