2tvenom/aggregator

Golang helper for parallel data processing

aggregator's Issues

Question / Aggregate data from Github api v3

Hi guys,

Hope you are all well !

I was wondering how I could be triggering some events cascading and async tasks with golang. I found your example and I am trying to connect the dots on the how to do that properly ^^

Use case:

1. Pitch: I want to manage my starred repository, and actually I use limo, written in go also.
2. Goal: Extend the update process, by fetching some additional informations from my starred repository like languages, the readme content and the file tree for the latest SHA.
3. Workflow:
   • Create a parent channel, with a concurrency of 20, like actually in the update.go file from limo, to fetch the list of my starred repo (per page: 100)
     • Fetch the readme (if exists), the git tree and the detected languages for each of them
     • Maybe 2 or 3 conditional requests per repository (eg. they have a wiki or more documents like additional markdowns)
4. Bottleneck:
   - How to configure this task of parallel and waterfall requests ?
   - How can I control that some missing readme, or some api requests errors won't break the integrity of my task-flow ?
   - What would describe the best my problem ?
   - Is it a much more a flow based programming (fbp) problem ?
   - Can I just use async commands by defining the flow within a simple yaml config ?

In fact, I would like to find or understand the easier approach to aggregate these data and not having un-optimized use of my goroutines.

More simply:
How can I re-use your example for such purpose ?

Cheers,
Richard