Persist Guardduty alerts to either Bigquery or GCS

[mozilla-services]

Persist Guardduty alerts to either Bigquery or GCS

[ajvb]

@adrianosela No, I think that it should be persisted outside of the context of foxsec-pipeline alerts. To be more explicit, we should persist the com.amazonaws.services.guardduty.model.Finding parsed within

foxsec-pipeline/src/main/java/com/mozilla/secops/parser/GuardDuty.java

Lines 11 to 22 in 49e28bf

	public class GuardDuty extends PayloadBase implements Serializable {
	private static final long serialVersionUID = 1L;
	private static final String CLOUDWATCH_EVENT_SOURCE = "aws.guardduty";
	private static ObjectMapper mapper =
	new ObjectMapper()
	// our input JSON may have properties not represented in {@link Finding} e.g. ignore
	// errors such as 'Unrecognized field "affectedResources"....'
	.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
	private Finding gdf;

to it's own BigQuery table or to GCS.

[ajvb]

Let's just do Guardduty. For ETD, we can use CSCC.

I'd base it off of https://github.com/mozilla-services/foxsec-pipeline/blob/master/src/main/java/com/mozilla/secops/OutputOptions.java#L23-L26 and https://github.com/mozilla-services/foxsec-pipeline/blob/master/src/main/java/com/mozilla/secops/CompositeOutput.java#L88-L107

So you should probably just be able to use org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO for this.

[adrianosela]

The "payload" field in my alerts is still empty (null), would this be a good place to throw the raw events? @ajvb

cc:// @ameihm0912

[adrianosela]

Gotcha, will implement a "Persistor" transform for each of ETD and GD. Will take the target BQ as an argument i.e. --persistETDFindingsBigQuery and --persistGDFindingsBigQuery and store them separately.

Does this sound like a reasonable approach? @ajvb @ameihm0912

Also, what should the behavior be if the user fails to provide these arguments? Run anyways and not persist, or die on deploy?

[adrianosela]

Closing in favor of #229 @ajvb