Feature Request: Lookup table Exclusion list or Exclusion Data Adapter

[ion-storm]

Feature Request: Lookup table Exclusion list or Exclusion Data Adapter

[joschi]

@ion-storm I'm not sure that's necessary or desirable.

Given we had such an "exclusion list" for each data adapter, where would it pull data from? An inline list? A CSV file? Any other data adapter?

So in the end you can already express such an "exclusion adapter" in a pipeline rule.

Example:

rule "lookup-with-exclusion"
when
  // some condition [...]
  // check if "exclusion_detail" is in "exclusion-lookup" lookup table
  && is_null(lookup_value("exclusion-lookup", to_string($message.exclusion_detail)))
then
  // Do whatever lookup you want
end

[ion-storm]

For example with the otx lookup we get threat alerts and some of them are listed under trackers at otx. It would be nice to have a exclusion list where we can list out exclusions to prevent a lookup for said hosts.

Also I'm seeing in debug logs that blah.systems.local is being looked up at otx, it would be nice to exclude our internal domain name or custom ip ranges. For now I'm having to add a stream rule for a few dozen streams that excludes it from alerting. Adding exclusions would be better, because over time a few hundred excluded IP's would be cumbersome to manage

[joschi]

@ion-storm As demonstrated in my previous reply, you can already do that in the when clause of a pipeline rule by querying a custom lookup table (e. g. based on a CSV file with your internal domains or IP ranges).

[ion-storm]

I attempted to add exclusions to threat-lookups, and this is failing, any ideas why, here is my pipeline:

rule "sysmon threatintel"
when
   // To save CPU cycles, only run if there is something to look up
   has_field("sysmon_dns_lookup") OR has_field("sysmon_dns_lookup_ip") OR has_field("sysmon_src_ip")
   && is_null(lookup_value("threat-exclusions", to_string($message.sysmon_dns_lookup)))
   //&& is_null(lookup_value("threat-exclusions", to_string($message.sysmon_dns_lookup_ip)))
   //&& is_null(lookup_value("threat-exclusions", to_string($message.sysmon_src_ip)))
then
    // look up the requested DNS captured by sysmon
    // this will be the most fired rule
    //let sysmon_dns_lookup_intel = threat_intel_lookup_domain(to_string($message.sysmon_dns_lookup), "sysmon_dns_lookup");
    //set_fields(sysmon_dns_lookup_intel);
    
    // look up the ip from the DNS answer
    // if we do not monitor the dns, then this might be nice to have
    //let sysmon_lookup_ip_answer_intel = threat_intel_lookup_ip(to_string($message.sysmon_dns_lookup_ip), "sysmon_dns_lookup_ip");
    //set_fields(sysmon_lookup_ip_answer_intel);


    // look up the requesting IP 
    // this is useful if dealing with non internal IPs 
    // so you know if your IP is seen as a problem
    //let sysmon_src_ip_answer_intel = threat_intel_lookup_ip(to_string($message.sysmon_src_ip), "sysmon_src_ip");
    //set_fields(sysmon_src_ip_answer_intel);

    //AlienVault OTX
    let intel = otx_lookup_ip(to_string($message.sysmon_src_ip));
    let intel = otx_lookup_domain(to_string($message.sysmon_dns_lookup));
    set_field("otx_threat_indicated", intel.otx_threat_indicated);
    set_field("otx_threat_ids", intel.otx_threat_ids);
    set_field("otx_threat_names", intel.otx_threat_names);
    
    //Tor Exit Nodes
    //let intel = tor_lookup(to_string($message.sysmon_src_ip));
    //let intel = tor_lookup(to_string($message.sysmon_dns_lookup_ip));
    //set_field("tor_threat_indicated", intel.threat_indicated);
    
    //Spamhaus
    let intel = spamhaus_lookup_ip(to_string($message.sysmon_src_ip));
    set_field("spamhaus_threat_indicated", intel.threat_indicated);
    
    //Abuse.ch Domain
    let intel = abusech_ransom_lookup_domain(to_string($message.sysmon_dns_lookup));
    let intel = abusech_ransom_lookup_ip(to_string($message.sysmon_dns_lookup_ip));
    set_field("request_domain_is_ransomware", intel.threat_indicated);
    
     // WHOIS lookup. This is disabled by default. Enable and carefully watch latency and performance.
    let sysmon_dns_lookup_ip_whois = whois_lookup_ip(to_string($message.sysmon_dns_lookup_ip), "sysmon_dns_lookup_ip");
    set_fields(sysmon_dns_lookup_ip_whois);
end

[ion-storm]

also rs6.net returns "na" like I want it to, and its not null

{
  "single_value": "na",
  "multi_value": {
    "value": "na"
  },
  "ttl": 9223372036854776000,
  "empty": false
}

[ion-storm]

so in this case, I want it to not lookup rs6.net but it does anyways

[dennisoelkers]

@ion-storm, it seems like you are trying to solve a different issue than the original report suggests. If you feel like pipelines has a bug, please open an issue in the corresponding repo.