Merge output from multiple servers

[dukexar]

Merge output from multiple servers

[GoogleCodeExporter]

I agree that there's often redundancy in the output, but I don't think there is 
any reasonable general-purpose way to address this.  However, it sounds like 
something like this would be helpful in a lot of special-case situations.  If I 
were trying to implement this, I think I would write a wrapper script that 
behaved roughly as follows:

1) Make a temporary directory (say $tmpdir).
2) Run pssh with -o and -e pointing to $tmpdir/out and $tmpdir/err.
3) When pssh terminates, run the case-specific comparison/consolidation code.
4) Output the summary.

I think you'll find that this isn't too hard to write, and it would be easy to 
customize for your specific needs.  If you think that this would be helpful to 
include with pssh, I would be happy to add it to a "contrib" directory with 
pssh.

Original comment by [email protected] on 7 Mar 2011 at 5:07

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

A long time ago, in a galaxy far away, I wrote a tool called ... pssh.  It did 
very very similar things to this tool, except it was written in Perl.  One of 
the things its users loved the most was precisely its ability to merge output 
from multiple servers.  The algorithm I used was the obvious one: do a search 
in each host's output for per-host data (i.e. hostname and IP address) and 
replace them with generic stubs ("[hostname]" and "[IP]" respectively).  Then 
calculate the MD5 digest of each munged output, group them according to this 
digest, and display each group of output, clearly indicating at the top of each 
which hosts belong to that group.

The output was collected in memory rather than on disk.  From a scalability 
perspective this was perhaps not the best approach, although in practice it was 
never a problem even when running across hundreds of machines.  Perhaps this 
time round I would go for on-disk collection, but on the strict condition that 
the tool should not *require* users to (a) think of a suitable output directory 
and (b) have to delete it after each run.  In other words it should default to 
a safely generated temporary directory, and default to cleaning up that 
directory unless the user explicitly specifies an output directory or 
explicitly asks for it not to be cleaned up.  This is because users want to be 
able to do multiple lightning quick comparisons across machines (e.g. pssh ... 
cat /etc/hosts), so any given pssh should not require extra commands (such as 
rm -rf $tmpdir) to be run afterwards.  But on the flip side, the user *will* 
sometimes want to grep -r through the output directory, so --no-cleanup should 
be an option when no output directory is explicitly specified.

I would recommend that output grouping be implemented as part of the core code 
rather than as a wrapper.  It is simply another output format, and so should be 
available via a CLI option.

BTW I have fallen out of love with Perl and returned from that galaxy far away. 
 So needless to say, I would also love to see this feature implemented by this 
project.

Original comment by [email protected] on 3 Mar 2012 at 11:34

[GoogleCodeExporter]

adam.spiers, the output grouping idea sounds pretty cool to me. I probably 
don't have time to work on this right now, but if you're interested in working 
on it, I'd be happy to answer any questions you have. With the "-i" option, 
we're already storing all of the output in RAM, and in practice, I don't think 
this would be the biggest scalability problem to worry about.

Anyway, I think it's a nice idea, and it wouldn't be all that hard to 
implement. If you've fallen out of love with Perl, it might be a ripe time to 
fall in love with Python. :)

Original comment by [email protected] on 3 Mar 2012 at 4:13

[GoogleCodeExporter]

I don't have time to work on it either, unfortunately :-/  I sent you an email 
with more details though :)

Actually I fell in love with Ruby way back in 2002.  I've had many flings with 
Python since then, and I'm quite fond of it, but it will never develop into a 
serious relationship after experiencing Ruby ;-)

Original comment by [email protected] on 3 Mar 2012 at 4:17