mozilla-services / foxsec-pipeline Goto Github PK
View Code? Open in Web Editor NEWLog analysis pipeline utilizing Apache Beam
License: Mozilla Public License 2.0
Log analysis pipeline utilizing Apache Beam
License: Mozilla Public License 2.0
#305 adds timeouts to the iprepd writer IO transform, but they are hardcoded. These should be configurable with reasonable defaults.
#289 modifies the windowing strategy used in Customs, but makes use of a default trigger.
This should be further enhanced with early pane firings, which could include potentially dealing with suppressing the on-time pane if needed to prevent duplicates.
See also comment in #181
Some elements of the pipeline put a list of values under a metadata key, right now it is essentially free form strings and up to the producer to properly delimit the elements.
That can be improved so there is a consistent method for putting a list of values into metadata.
It would be nice to be able to view metrics on how our code coverage is doing across our unit tests. If we have a tool that can support multiple languages to cover contrib
, that would be nice, but making sure can see the % coverage of the Java code is the more important piece.
The existing identity class can lookup for example a global user ID given aliases
It would be good to expand this for other lookups, for example AWS account IDs to a descriptive name to contain this in a single configuration.
https://github.com/mozilla-services/foxsec-pipeline-contrib contains a few cloudfunctions that it would be nice to include within the integration tests in this repo.
Since Cloudfunctions written in Go are just libraries, we could create a simple mock service that simulates both data ingestions (Duopull/Auth0pull) and interacting with the pipeline (SlackbotBackground).
Would be especially useful for testing parsing and data model changes.
Rather than pass an encrypted string via pipeline options, pass a GCS path that will contain the path to the encrypted string
In addition to the UTC timestamp, also include the localized timestamp based on the source address in the alert metadata and the alert payload (e.g., render it in the template/Slack notifications)
#174 adds a pipeline for monitoring output from ETD and GD.
In the current state, the pipeline will generate alerts but it does not currently add any escalation metadata to the alerts that will result in special handling in AlertIO
.
Example from the authentication pipeline:
foxsec-pipeline/src/main/java/com/mozilla/secops/authprofile/AuthProfile.java
Lines 255 to 263 in 7a27d94
We will want to add a similar notification pipeline option to the gatekeeper pipeline and include this metadata option when we generate an alert if the option is set in the configuration, if the option is not set the pipeline will behave as it currently does.
We will also want to potentially look at hooking AlertSuppressor
up to the end of the analysis transforms and suppressing possible repeated alerts to avoid generating a large number of escalation notifications under certain circumstances.
It would be nice to include the AWS account name / the GCP project name in the Gatekeeper alerts for easier triaging.
For AWS, the IdentityManager can be used to translate the account id to account name:
For GCP, projects.get
(or similar) can be used during alert creation to grab the project's metadata.
Add EC2 instance ID, owner, vpc, etc to GuardDuty alerts.
This data is already available within the Finding objects
Add EC2 instance ID, etc to GuardDuty alerts.
This data is already available within the Finding objects
The current version of the AuthProfile pipeline only uses email templates for a new source, but does not use it for informational alerts.
To be consistent it would be good to just support this across the board in the pipeline, but it would require an additional template.
See also #23
As an example, have a link to https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_stealth.html within the alert body (https://bugzilla.mozilla.org/show_bug.cgi?id=1626813#c35) since the finding type is Stealth:IAMUser/CloudTrailLoggingDisabled
Currently, we are only tagging gatekeeper alerts with the base metadata on guardduty findings. Some finding types contain a lot more relevant data.
For example, the ssh brute force finding type and recon types contain an "Actor" object, which contains the bad actor IP, location (latitude + longitude), the ISP, etc.
This issue encompasses adding metadata on a per-finding-type basis to alerts
So far Guard Duty has been pretty good at classifying ALL network connection type (brute force attempts, port scans, etc) as low severity.
However, it generates many high severity "DNS REQUEST" type findings for EC2 instances querying bad domains (crypto, phishing, malware, etc).
These are NOT false positives, but in most cases we don't care about them because they are expected due to the nature of the running host.
For example: we run a ton of web crawlers to asses the state of the web in general, etc, these hit those bad domains all the time, and we dont care.
We should have a way of suppressing these type of alerts for these EC2 instances. That should not be hard given the fact that the ec2 instance tags are contained in the finding object.
CC:// @ameihm0912 @ajvb
We aren't persisting anything outside of the built-in persistence options of the data sources we read from.
Fulfilling this issue involves storing all generic events somewhere
As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:
If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].
(Message COC001)
The parser currently only supports processing nginx log data in the form of a Stackdriver jsonPayload entry. This should be expanded to also support raw nginx log lines (either in textPayload or on their own).
If a particular transform in Customs
has escalation disabled, include (experimental)
at the end of the transform doc string
Provide a link with notification that results in a query to pull up applicable alerts.
Several customs transforms should be refined with a variance component including:
CustomsPasswordResetAbuse
CustomsAccountCreation
SourceLoginFailure
to work around issues associated with large residential/mobile NAT pools.
Style Guide: https://google.github.io/styleguide/javaguide.html
CLI tool: https://github.com/google/google-java-format
Vim plugin: https://github.com/google/vim-codefmt
I have no strong feelings about this style guide, but it would be nice to be able to use the vim plugin to support automatic and consistent formatting.
Currently, the code the checks if an entry has been whitelisted for iprepd only works with IP addresses; this should be expanded so it works with other types as well (e.g., email).
Depends on mozilla-services/foxsec-pipeline-contrib#21
The alerts are purely based on exceeding a particular distance over time right now.
The toggle would allow for an optional minimum distance to for example avoid creating alerts if the velocity is exceeded and the distance was only 50km.
Where addresses within alerts can be reasonably correlated to belonging to the same subnet, for example the same /24, add support to alerting output to potentially generate a secondary alert indicating a probable bad subnet.
This would essentially be something like a reduction operation to output a subnet given an set of input elements.
AuthProfile
currently requires being restarted to reload a new identity manager configuration; this could be improved by setting a per-thread timer and reloading within @ProcessElement
after a certain expiration time.
If known at the time, store UA in AuthStateModel.ModelEntry
for state storage
Currently cfgtick contains runtime options and transform documentation; it would be good in some cases to support the addition of arbitrary text blocks (potentially via configuration options) as well
When pipelines scale up, it will result in the creation of new worker nodes and additional threads calling setup
in certain DoFn
s. In some cases when the number of new workers is large, it can result in exceptions being generated in the pipeline due to quota limits being hit.
java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: com.google.api.gax.rpc.ResourceExhaustedException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'cloudkms.googleapis.com/crypto_requests' and limit 'CryptoRequestsPerMinutePerProject' of service 'cloudkms.googleapis.com' for consumer
Although it generally recovers eventually this causes errors to be propagated by the pipeline and likely slows down scaling responsiveness.
As we discussed, it would be helpful to persist all Guardduty alerts in there raw json to either a separate BigQuery table or to GCS.
The goal is to be able to dig into alerts easily during an investigation.
Currently events can be normalized into a single category (e.g., AUTH), but in some cases a single event may represent more than one category type.
Ideal scenario would be the normalized category field being a bitmask or array of values that can represent more than one type.
New pipeline, intended to:
Alert
JSON strings)Initially it would be ideal for this to consume a topic containing Alert
JSON strings, parse them into Alert
objects, and push these objects into an analysis step.
The analysis step will look for metadata fields that for example match a given regex, and where a match occurs a new alert will be generated. This alert will contain a metadata field containing the ID of the source alert which it matched on.
This new pipeline will also be useful for future correlation between pipeline alerting output.
There is code in IprepdIO
to support the transition to a new Datastore format for whitelisted objects, i.e.:
foxsec-pipeline/src/main/java/com/mozilla/secops/IprepdIO.java
Lines 584 to 588 in 56045c4
This can be removed now
Make AlertMeta more strictly defined rather than the current free form key strings everywhere.
Currently there are free form ("magic strings") everywhere for metedata key names. This is also present throughout the cloud functions (https://github.com/mozilla-services/foxsec-pipeline/blob/master/contrib/bugzilla-alert-manager/manager.go#L121).
We should make these keys more strictly defined and do this in a way that can be tested and checked for both the Java and Go code.
Related to #182
If the function is updating a closed bug with new information, it should be reopened as part of the update
The output path of the pipelines is currently limited to ingestion of Alert
objects.
This makes it difficult to persist other types of data from the pipeline that are not neccessarily alerts but we may want to write to BigQuery. This is discussed a bit in #320.
The intent would be to modify the output path to consume this new container event, which could contain an alert or some other type of object (like a source summary, etc). Alert
objects would be handled the way way they currently are once the encapsulation is stripped, but this would provide a means to produce other types of pipeline output.
IprepdIO currently only supports submission to a single instance
This should be modified so multiple iprepd instances can be configured, and the transforms will publish to all configured instances
API: https://help.papertrailapp.com/kb/how-it-works/search-api/
Some past work in this area (Lua): https://github.com/mozilla-services/lua_sandbox_extensions/tree/master/papertrail/sandboxes/heka/input
The exception filtering currently filters any request of a given method and path; extend this to only filter if the request is either 200, or >= 500, but do not filter 4xx errors.
The maven docker image version was pinned in 9c50679 to work around an openjdk bug introduced recently related with relative classpath loading that causes surefire to crash
See also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911925
A workaround is available, but rather then work around it for now this should be reverted once a fixed image is available.
We should write tooling to support integration tests between the cloud functions in contrib/ and the pipeline code.
Some examples:
There is currently not emulator tool for Cloud Functions written in Go, but these functions are written as a library, so wiring up our own runner for them should be easy enough (this is also how some of the unit tests work for the cloud functions).
The alerts currently only store the new location in the metadata. Modify to also store the previous location.
Within our parsing logic, we make use of the pubsub timestamp rather than the parsed events timestamp (Parser.stripStackdriverEncapsulation
). This creates problems in the case of old messages getting backfilled into pubsub for whatever reason and causes a disconnect between our tests and production.
Instead, we should make use of the parsed out event timestamp.
Geolocation metadata could be useful for running analysis of http_request
alerts.
Currently only a description of the type of event is provided. It would be good if this contained some information specific to the event itself as well (e.g., username if applicable, etc).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.