Keep local server register of started & completed segments

[hazyresearch]

Keep local server register of started & completed segments

[ajratner]

Simplest / quickest way I can think of doing- parser writes a .reg file which is "0" when started and running and "1" when completed. fab can simply collect all the .reg files- super simple- and then come up with register of global status

[ajratner]

@chrismre Have to run to dinner with Will (timeliness!) but quick rundown of process (bold is what I'm adding):

1. Files split locally into segments
2. Segments are distributed across servers / cores
3. When a core picks up a segment, it writes "0" to seg_id.reg in addition to writing output to seg_id.parsed
4. When a core completes a segment it writes "1" to seg_id.reg
5. Global status can be checked at any time via a simple 'collect' operation over *.reg in fab
6. At the end, the *.parsed files are collected and catted to get full output...

Any thoughts? This just seemed quick and simple to implement given the current setup

@raphaelhoffmann any thoughts?

[raphaelhoffmann]

That is really cool! I like that you can do a global status check with a
simple command.
Thanks for building this.

On Thu, Aug 6, 2015 at 5:57 PM, Alex Ratner [email protected]
wrote:

@chrismre https://github.com/chrismre Have to run to dinner with Will
(timeliness!) but quick rundown of process:

1. Files split locally into segments
2. Segments are distributed across servers / cores
3. When a core picks up a segment, it writes "0" to seg_id.reg in
   addition to writing output to seg_id.parsed
4. When a core completes a segment it writes "1" to seg_id.reg
5. Global status can be checked at any time via a simple 'collect'
   operation over *.reg in fab
6. At the end, the *.parsed files are collected and catted to get full
   output...

Any thoughts? This just seemed quick and simple to implement given the
current setup

—
Reply to this email directly or view it on GitHub
#13 (comment).

[ajratner]

@raphaelhoffmann Just need to merge my jsonreader-multi-key branch into master, or the relevant part of it, for this to work