Archive function does not persist the "archived" state

[satta]

Archive function does not persist the "archived" state

[jasonish]

Should be fixed now. 0.17.1 pushed.

[jasonish]

I just re-tested on 8.5.3 and can't reproduce (most of my testing has been on 8.6.0 lately). Can you tell me how you are adding the events? Any custom mappings?

[satta]

Sure! This is on my private personal setup; events are received on the sensor side via syslog-ng and sent to a RabbitMQ exchange; on the Elasticsearch side there is a Logstash instance collecting data from there:

input {
    rabbitmq {
        host => "rabbit...."
        port => 5672
        heartbeat => 30
        exchange => "alerts"
        exchange_type => "fanout"
        durable => true
        queue => "alerts.elastic"
        user => "consumer"
        password => "consumer"
    }
}
output {
    elasticsearch {
        hosts => "localhost:9200"
        index => "logstash-%{+yyyy.MM.dd}"
    }
}

Here's an example event I triggered to test the alerting path, just copied from Evebox's "JSON" box (which isn't really valid JSON 🤔):

{
   _id: "9Pe3GIcB2gitsSQ5lBbt",
   _ignored: [
      "event.original.keyword",
      "alert.rule.keyword"
   ],
   _index: "logstash-2023.03.25",
   _metadata: {
      aggregate: true,
      count: 2,
      escalated_count: 0,
      max_timestamp: "2023-03-25T12:21:36.888083+0000",
      min_timestamp: "2023-03-25T12:21:36.811383+0000"
   },
   _score: null,
   _source: {
      "@timestamp": "2023-03-25T12:21:37.029914885Z",
      "@version": "1",
      alert: {
         action: "allowed",
         category: "Potentially Bad Traffic",
         gid: 1,
         metadata: {
            created_at: [
               "2012_01_31"
            ],
            updated_at: [
               "2020_09_14"
            ]
         },
         rev: 4,
         rule: "alert dns $HOME_NET any -> any any (msg:\"ET DNS Query for .su TLD (Soviet Union) Often Malware Related\"; dns.query; content:\".su\"; nocase; endswith; reference:url,www.abuse.ch/?p=3581; classtype:bad-unknown; sid:2014169; rev:4; metadata:created_at 2012_01_31, updated_at 2020_09_14;)",
         severity: 2,
         signature: "ET DNS Query for .su TLD (Soviet Union) Often Malware Related",
         signature_id: 2014169
      },
      app_proto: "dns",
      dest_ip: "8.8.8.8",
      dest_port: 53,
      dns: {
         query: [
            {
               id: 64021,
               rrname: "foo.su",
               rrtype: "AAAA",
               tx_id: 0,
               type: "query"
            }
         ]
      },
      event: {
         original: "{\"timestamp\":\"2023-03-25T12:21:36.888083+0000\",\"flow_id\":817065574305043,\"in_iface\":\"br0\",\"event_type\":\"alert\",\"src_ip\":\"192.168.178.25\",\"src_port\":56071,\"dest_ip\":\"8.8.8.8\",\"dest_port\":53,\"proto\":\"UDP\",\"tx_id\":0,\"alert\":{\"action\":\"allowed\",\"gid\":1,\"signature_id\":2014169,\"rev\":4,\"signature\":\"ET DNS Query for .su TLD (Soviet Union) Often Malware Related\",\"category\":\"Potentially Bad Traffic\",\"severity\":2,\"metadata\":{\"created_at\":[\"2012_01_31\"],\"updated_at\":[\"2020_09_14\"]},\"rule\":\"alert dns $HOME_NET any -> any any (msg:\\\"ET DNS Query for .su TLD (Soviet Union) Often Malware Related\\\"; dns.query; content:\\\".su\\\"; nocase; endswith; reference:url,www.abuse.ch/?p=3581; classtype:bad-unknown; sid:2014169; rev:4; metadata:created_at 2012_01_31, updated_at 2020_09_14;)\"},\"dns\":{\"query\":[{\"type\":\"query\",\"id\":64021,\"rrname\":\"foo.su\",\"rrtype\":\"AAAA\",\"tx_id\":0}]},\"app_proto\":\"dns\",\"flow\":{\"pkts_toserver\":1,\"pkts_toclient\":0,\"bytes_toserver\":66,\"bytes_toclient\":0,\"start\":\"2023-03-25T12:21:36.888083+0000\"},\"payload\":\"+hUBAAABAAAAAAAAA2ZvbwJzdQAAHAAB\",\"stream\":0,\"packet\":\"OBDV6YO4MCQyA+8NCABFAAA0VE4AAEARo5nAqLIZCAgICNsHADUAILwi+hUBAAABAAAAAAAAA2ZvbwJzdQAAHAAB\",\"packet_info\":{\"linktype\":1},\"src_host\":[{\"rdns\":\"BLN04NB0421.fritz.box\"}]}\n"
      },
      event_type: "alert",
      flow: {
         bytes_toclient: 0,
         bytes_toserver: 66,
         pkts_toclient: 0,
         pkts_toserver: 1,
         start: "2023-03-25T12:21:36.888083+0000"
      },
      flow_id: 817065574305043,
      in_iface: "br0",
      packet: "OBDV6YO4MCQyA+8NCABFAAA0VE4AAEARo5nAqLIZCAgICNsHADUAILwi+hUBAAABAAAAAAAAA2ZvbwJzdQAAHAAB",
      packet_info: {
         linktype: 1
      },
      payload: "+hUBAAABAAAAAAAAA2ZvbwJzdQAAHAAB",
      proto: "UDP",
      src_host: [
         {
            rdns: "redacted.internal.host.name"
         }
      ],
      src_ip: "192.168.178.25",
      src_port: 56071,
      stream: 0,
      timestamp: "2023-03-25T12:21:36.888083+0000",
      tx_id: 0
   },
   sort: [
      1679746897029
   ]
}

The only nonstandard thing I can think of is that FEVER (in between) does active reverse DNS queries to enrich internal IPs within the EVE with a src_host sub-object containing the hostname, to make it easier to tell what device in my home network triggered the alert. Not sure if that could be an issue.

[jasonish]

I've reproduced it. I have to admit its been sometime since I've done plain Logstash->Elasticsearch, probably 7.0 days. It appears they've changed the schema in 8. It still works in SELKS which uses Logstash, but they're still on ELK 7. This is something I'll need to fix ASAP.

As for the JSON, yeah, its valid Javascript not JSON I guess, as the keys are not quoted. Visually I guess it looks better unquoted, but I should probably make it valid.

[satta]

Thanks for looking into it!

[satta]

Works perfectly now. Thanks for addressing this quickly!