rfizzle/log-collector

Generic log collector with an assortment of sources

log-collector's Issues

V1: Proposal

Redesign with inputs -> processing pipeline -> outputs. Similar to the design of LogStash.

• Manager: the manager will handle all the instances of collectors.
• Instance: the object that will handle the state of a collector and the input, processors, and outputs.
• Input: generic interface for inputs.
• Processors: generic interface for processors.
• Pipeline: an ordered set of processors.
• Outputs: generic interface for outputs. Will be async and include error reporting.

input -> processor pipeline -> output will communicate via channels

Generic interfaces will take a json.RawMessage config and serialize as needed.

Configs will be located in individual files or configured via a custom JSON API pull with a refresh timer.

Each unique individual file will have its own instance state, to support multiple configs for the same input, and maintain separate states.

Should have the ability to kill and spawn instances without restarting the entire service (for JSON API dynamic config).

Draft Collector Instance Config Spec:

{
  "input": {
    "type": "gsuite_audit",
    "schedule": 3600,
    "config": {
      "credentials": "/path/to/key/file"
    }
  },
  "processors": [{
    "type": "replace"
    "config": {
      "search_value": "",
      "replace_value": ""
    }
  }],
  "outputs": [
    {"type": "stdout"}
  ]
}