Malware Archive not showing archived files + related issues & feedback

[kam193]

Malware Archive not showing archived files + related issues & feedback

[cccs-sgaron]

This should fix the issue: CybercentreCanada/assemblyline-ui-frontend#941

[kam193]

Thanks! Do you maybe know if it's expected to remove older archived submissions (after expiration), or it's just because the archive wasn't released yet?

[cccs-sgaron]

Older archived submissions should not be removed as they don't have any expiry_ts value. In the end the process that expires the data will expire anything that has an expiry_ts value at the time of the value. The archiving process explicitely clears that date.

[kam193]

Hey, this is what I expected - and unfortunately, not what I see. Please take a look:

ES query:

GET submission/_search
{
  "query": {
    "match": {"archived": true}
  },
  "_source": ["archived", "archive_ts", "metadata.ts", "expiry_ts"]
}

Response:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 13,
      "relation" : "eq"
    },
    "max_score" : 5.9746575,
    "hits" : [
      {
        "_index" : "submission_hot",
        "_type" : "_doc",
        "_id" : "7cXDHh2j2bJA8cSMrO2Iff",
        "_score" : 5.9746575,
        "_source" : {
          "archived" : true,
          "metadata" : {
            "ts" : "2024-02-08T04:11:50.392027Z"
          },
          "archive_ts" : null,
          "expiry_ts" : "2024-02-13T04:11:50.392183Z"
        }
      },    
      {
        "_index" : "submission_hot",
        "_type" : "_doc",
        "_id" : "7JnQYw67zLwmI6vb4MtPTt",
        "_score" : 5.9746575,
        "_source" : {
          "archived" : true,
          "metadata" : {
            "ts" : "2024-02-09T17:06:15.830477Z"
          },
          "archive_ts" : null,
          "expiry_ts" : "2024-02-14T17:06:15.830537Z"
        }
      },
[...]  
      {
        "_index" : "submission_hot",
        "_type" : "_doc",
        "_id" : "4QKReAE5AkIIyZaOjF9bJp",
        "_score" : 5.977854,
        "_source" : {
          "archived" : true,
          "archive_ts" : null,
          "expiry_ts" : "2024-02-17T19:11:38.466351Z"
        }
      }
    ]
  }
}

The first one was probably on the older version, but the last just a few minutes ago, and here is the log from archiver (version stable91):

{"@timestamp": "2024-02-12 19:14:12,085", "event": { "module": "assemblyline", "dataset": "assemblyline.archiver" }, "host": { "ip": "x.x.x.x", "hostname": "475fe3517330" }, "log": { "level": "INFO", "logger": "assemblyline.archiver" }, "process": { "pid": "1" }, "message": "Successfully archived submission '4QKReAE5AkIIyZaOjF9bJp'."}

As you can see, they all have the expiry_ts set. In addition, I haven't seen any archived file without expiry_ts, some archived submissions have also already been removed. Could it be that the configuration datastore.archive.indices is, in fact, empty by default, although the documentation says it's not?

[cccs-sgaron]

No this is working fine, you are just looking at the wrong index. The archiving process copies the data from submission_hot to submission-ma_hot. All documents in submission_hot will still have their expiry_ts but documents in submission-ma_hot will not. From what I see in your results, all the documents that you found with an expiry_ts are in the non -ma indices.

[kam193]

Ah, right, thanks for the explanation! I should have verified this first