cybercentrecanada/assemblyline

AssemblyLine 4: File triage and malware analysis

What is the purpose of this repo?

This is a repository containing development resources for the Assembyline project.

"A scalable file triage and malware analysis system integrating the cyber security community's best tools!"

Documentation: https://cybercentrecanada.github.io/assemblyline4_docs/

---

"Plateforme de tri de fichier et d'analyse de « malware » qui intègre les meilleurs outils de la communauté en cyber sécurité!"

Documentation: https://cybercentrecanada.github.io/assemblyline4_docs/fr/

Core Components

Repository Name	Main Branch	Dev Branch	PyPI Release
Assemblyline Base
Assemblyline Core
Assemblyline Service Client
Assemblyline Service Server
Assemblyline UI
Assemblyline UI Frontend			N/A
Assemblyline Service Base

Auxiliary Components

Repository Name	PyPI Release
Assemblyline Client
Assemblyline Service Utilities
Assemblyline Incident Manager

assemblyline's Issues

VMRay Service

Is your feature request related to a problem? Please describe.
Has been requested over the years but there is no official service because of the license requirements & lack of API documentation:
VMRay integration (google.com)

Describe the solution you'd like
Community-written service to leverage VMRay analysis

Feature Request: Submissions should allow tagging

Is your feature request related to a problem? Please describe.
Assemblyline should have the ability to add tags to a submissions, this can be used to tag campaigns, groups, and more.

Describe the solution you'd like
Each submission should have the ability to be tagged by the analyst, such that the list would appear somewhere along here:

users should than be able to search based on these tags, and also query the API based on these tags.

These tags are not the IOC tags extracted from the submission, here tags refers to a free form text which would allow us add extra information to the submission.

One use case would be tagging a submission as "production" this means that automation can pick the IOCs up from this submission and push them to production.

Another user case would be tagging a submission with a particular APT name, users can than query or search on these tags to find all the malware samples belonging to that APT, similar use case for campaigns.

There should not be a limit to how many tags a submission can have, or maybe controlled through values.yaml

Describe alternatives you've considered
n/a

Additional context
n/a

Feature Request - Deploy analytic service updates via helm chart

The basic proposal is to allow analytic services to be updated via the helm chart.

You can already install the analytic service via the helm chart when specifying the REGISTER_ONLY env var for a given service as the current AL4 helm chart already does. This would take it a step further and allow for service updates. It would allow you to specify an updated service_manifest.yml along with a new version tag. This can already be done in the existing system if you simply re-run the service install k8s job and specify an updated service_manifest. The change requested, is when a service is updated (via a k8s job for example), the new service version merges and replaces the existing version. The expected behavior can be replicated by calling the assemblyline python client service.add() followed by service.update().

This is a git ops focused approach allowing the current config of the system to be managed in git.

Binary Retention

Currently, the retention policy is set by the user in the Days to live. We can set a default, but we can't override that setting. Our org's intention is to not delete the binaries. We would like to set a system retention policy so that we may later use those binaries for further analysis, retro-hunting, etc. If report storage / performance is a concern, perhaps the report could apply to Days to live but the binary remains along with a record of the user's submission.

Enhancement: Try 4 offsets for potential base64 encoded strings

If the boundary of the matched base64 string does not align with the original offset, the expected strings will not be found.

For example, .exePK\x01\x02 can be extracted from each of these strings, but this service will only find that on the first string because the matched offset aligns original offset.

    LmV4ZVBLAQI 
   0LmV4ZVBLAQI
  J0LmV4ZVBLAQI
 3J0LmV4ZVBLAQI

BBCrack Errors in Python3 vs Python2

https://github.com/CybercentreCanada/assemblyline-v4-service/blob/8aa77e6f1ee0c943ba9ff8710edf5ff19c130d80/assemblyline_v4_service/common/balbuzard/bbcrack.py#L773

If a service imports bbcrack and passes it a level not equal to 1, 2 or small_string, then the Level 3 transforms will be attempted.
https://github.com/CybercentreCanada/assemblyline-v4-service/blob/8aa77e6f1ee0c943ba9ff8710edf5ff19c130d80/assemblyline_v4_service/common/balbuzard/bbcrack.py#L730

Slices of bytestrings returns either the ordinal representation of the slice in Python3, or the bytestring in Python2. Since we are still trying to call ord() in these level 3 transforms, despite using Python3 in AL services this will throw an error.

Not sure if any services are using this level, but if they did the error would be raised.

FIPS mode fails to import Python 3.9 crypt

Describe the bug
If kernel FIPS mode is enabled, the UI and services fails because they cannot import crypt.

To Reproduce
Steps to reproduce the behavior:

1. Enable kernel FIPS mode
2. Attempt to start development environment with Docker Compose
3. UI fails to start and logs show a crypt import error

Expected behavior
Import crypt statement succeeds

Additional context
This issue is fixed in Python 3.10 and higher.
python/cpython@2fa03b1
python/cpython#95231

[feature request] related submissions

Hi,

when inspecting a submission, it would be helpful to have the list of related submissions.
And for each file of the submission, add the list of related submissions as well.

For example :

Submission_A : file1, file2, file3
Submission_B : file2
Submission_C : file3

related submissions of Submission_A : [Submission_B, Submission_C]
related submissions of file2 : [Submission_A, Submission_B]

This way an analyst can easily see if a file has already been seen elsewhere and quickly map the relationships.

This feature would impact the endpoint /api/v4/submission/report/<sid>/ and the front-end I believe :

GET /api/v4/submission/report/<submission_id_A>/
{
    "api_response": {
        ...,
        "related_submissions": ["<sid_B>", "<sid_C>"],
        "file_tree": {
            "sha256_file2": {
                "related_submissions": ["<sid_B>"],
                ...
            }
        }
    }
}

or maybe another endpoint should be created.

Thank's !

Feature Request: Scaler should allow windows nodes to be orchestrated

We have an ever increasing usecase to run windows services, as it stands we are communicating with external environments to run windows binaries.

Describe the solution you'd like
Assemblyline, should allow us to orchestrate windows containers as well as Linux containers, this will centralise the management of resources to just the k8s cluster instead of setting up and managing auxiliary infrastructure, which does not necessarily scale as well.

Describe alternatives you've considered
Created VMs with an API server to receive process and respond to Assemblyline

Additional context
n/a

Feature Request: Make the carousel images part of the URL to make it easier to share with others

Is your feature request related to a problem? Please describe.
We want to make the carousel image like part of the URL so when a user shares the link, it will open the image. However, we do not want it to part of the history.

Enhancement: Services Loaded

It would be nice if you could add a column for Internal / External services. That would be great to see at a glance since it involves less network isolation and sending info about your malware to others (even if it's just a hash).

processing for NSIS installers

Discord: https://discord.com/channels/908084610158714900/908717441771794472/1019296110239567963

Does Extract (or any other service) have any processing for NSIS installers? Was hoping to extract [NSIS].nsi if it's available in the exe file

Possible decompilers: https://nsis.sourceforge.io/Can_I_decompile_an_existing_installer%3F
Magic for identify: Nullsoft Installer self-extracting archive

Update to TLP 2.0

Please update to the TLP version to 2.0.

TLP version 2.0 is the current version of TLP standardized by FIRST. It is authoritative from August 2022 onwards.
https://www.first.org/tlp/

Feature Request: Submitting URL or IP should allow users to pick country/region the proxy will egress from.

Is your feature request related to a problem? Please describe.
When submitting a domain or IP, users should be allowed to pick what country region to egress out of, maybe a general setting in the UI configurable per user would be a way to do this.

Describe the solution you'd like
In some cases we want to egress out of a specific country when submitting a domain or IP in Assemblyline, the only way to do this at the moment is to update the backend proxy settings, we should be able to do this through the UI.

Describe alternatives you've considered
Update values.yaml manually.

Additional context
N/A

YARA UI & Service Updates

1. It would be nice if the Signature Search also searched on Source. So if I search bartblaze or reversinglabs, I'd get some Yara hits for Signatures. I couldn't find the Yara (perhaps due to a 10,000 limit on Signatures) so I was searching on the Source and kept getting 0 hits. I didn't realize I needed to search Yara. I did end up finding them by clicking on the source fingerprint.

2. Some of my yara source updates don't appear to be processing the yara rules. Here are couple public repositories that I've added that say they've downloaded, but show no signatures.

   • (.*.yara$) https://github.com/chronicle/GCTI.git
   • (.*.yara$) https://github.com/reversinglabs/reversinglabs-yara-rules.git

API Keys Shown in results.json

When a submission bundle is downloaded, the results.json file can include user VirusTotal API Keys.

Steps to reproduce the behavior:

1. Submit file with VirusTotal Service API Key
2. Download submission bundle
3. Extract results.json
4. API Key is visible

API Keys should not be visible in shared output.

• Assemblyline Version: 4.4.0

Other potential related issues:

• Exposing another user's API Key when resubmitting a file with the same submission options
• Exposing service level API Keys in submission bundles

Debugging use of invalid API Keys may be more difficult if they are removed entirely.

Extract more files using hachoir-subfile (or other tools)

Example hash:
c7dd490adb297b7f529950778b5a426e8068ea2df58be5d8fd49fe55b5331e28

hachoir-subfile output:

# hachoir-subfile c7dd490adb297b7f529950778b5a426e8068ea2df58be5d8fd49fe55b5331e28.doc
[+] Start search on 1563136 bytes (1.5 MB)

[+] File at 0 size=1563136 (1.5 MB): Microsoft Office document
[+] File at 512: Microsoft Office Word document
[+] File at 8775 size=1371772 (1.3 MB): PNG picture: 2480x3508x24

When running this same file through AL, the PNG file is not extracted. An extracted ole object contains the png file, but the image itself does not appear anywhere in the AL output.

Perhaps this could be a deep scan feature, since it may add a lot of artifacts that people may not care much about.

Enhancement: Add Wide search to Hex Viewer

The ability to search for UTF16 strings like "http" could be simpler if there was a "wide" option.

YARA 4.3.0 will break `_add_string_match_data`

Is your feature request related to a problem? Please describe.
This release [4.3.0] introduces backward incompatible changes to the yara-python API.

In particular, it will impact: https://github.com/CybercentreCanada/assemblyline-service-yara/blob/3061ac084004db4cb514d192ab88fd142ad9d09e/yara_/yara_.py#L225

Describe the solution you'd like
Update service code to account for this breaking change.

Minio does not support being backed by host FS anymore

Since the release of RELEASE.2022-10-29T06-21-33Z minio does not support being backed by the host file system. If an admin would chose to change the compose file to bind a host FS to the minio "/data", the following error is triggered when trying to run AL when the latest version of MINIO container is used.
ERROR Unable to use the drive /data: Drive /data: found backend type fs, expected xl or xl-single
Documentation Reference

As a workaround one could modify the compose file to use the last image that supports FS. Like this
image: ${REGISTRY}minio/minio:RELEASE.2022-10-24T18-35-07Z

It might be a good idea to add a note in the documentation and have a way to support host FS, specially since the filestore would usually run on a slower disk.

Getting error from a file submission using assemblyline-client 4.5.0

Describe the bug
Using the assemblyline-client, version 4.5.0 to submit a file for analysis, the client 'submit()' is returning an error: "Error: init() missing 1 required positional argument: 'status_code'".

In the assemblyline-client code, the 'ClientError' class takes a 'status_code' as a required parameter, and I think whatever is throwing an exception is trying to instantiate a 'ClientError', but failing to supply a 'status_code'. So, instead of seeing the actual error that originally threw the exception, we are getting an error message from the failed 'ClientError' instantiation.

To Reproduce
I don't think you could reproduce this without using the same file we are trying to submit.

Expected behavior
Something is going wrong with the file submission, and we would expect to get an informative error message showing the actual cause of the problem.

Environment (please complete the following information if pertinent):

• Assemblyline Version: [e.g. 4.3.0]
• assemblyline-client 4.5.0

missing "original_selected" field in notification queue

Users report that notification queue objects had a field "original_selected" set in the submission parameters which is now missing.

This was probably supplementary information being added by ingester for resubmits. Check if that was the case and if so, look into making sure the information is still available in some way when the notification is published by the new post-processing module.

TLSH fuzzy hash

TLSH is a fuzzy matching program and library. Given a file (min 50 bytes), TLSH generates a hash value which can be used for similarity comparisons. Similar files will have similar hash values which allows for the detection of similar objects by comparing their hash values TLSH has been adopted by a range of bodies and malware repositories including:

• VirusTotal (https://developers.virustotal.com/v3.0/reference#files-tlsh)
• Malware Bazaar (https://bazaar.abuse.ch/)
• MISP (https://www.misp-project.org/)
• STIX: TLSH is a part of the STIX 2.1 standard (https://docs.oasis-open.org/cti/stix/v2.1/stix-v2.1.html).

More information about TLSH, along with diff compare examples and reading in via buffer can be found at: https://tlsh.org/

Below is what we currently have for including TLSH as a Service, but we think TLSH would be a nice addition to the AssemblyLine Core to include with other hashes like SSDEEP. Installed via py-tlsh

import shutil
import tempfile
from pathlib import Path

import tlsh
from assemblyline_v4_service.common.base import ServiceBase
from assemblyline_v4_service.common.result import Result, ResultJSONSection


class TlshService(ServiceBase):

    def __init__(self, config=None):
        super(TlshService, self).__init__(config)

    def start(self):
        self.log.info(f"start() from {self.service_attributes.name} service called")

    def execute(self, request):
        self.log.info(f"execute() from {self.service_attributes.name} service called for '{request.file_name}'")

        try:
            with open(request.file_path, "rb") as sample_file_object:
                working_dir = Path(tempfile.mkdtemp())

                try:
                    sample_bytes = sample_file_object.read()
                    tlsh_digest = tlsh.hash(sample_bytes).lower()

                    result = Result()

                    json_section = ResultJSONSection("TLSH Response")
                    json_section.set_json({'tlsh': tlsh_digest})
                    result.add_section(json_section)

                    request.result = result
                finally:
                    shutil.rmtree(working_dir, ignore_errors=True)

            result = Result()

        except Exception as ex:
            self.log.error(str(ex))
            raise

Incorrect file.parent in ontology result

Describe the bug
The file.parent element in the ontology result points to the root file of the submission rather than the actual parent.

To Reproduce
Take a file that has child files extracted, and further children extracted from those files so you have a hierarchy 3 or more deep. When retrieving the ontology results for that submission via /api/v4/ontology/submission/<sid>, the file.parent element of a level 2 or more child file is specifying the root file hash of the submission rather than the actual parent of the extracted file.

Expected behavior
The file.parent element of an ontology result should point to the direct parent of that file.

Diagram showing Containers, DBs and their interaction

Hi, we would like to know if you have available Diagrams showing the Containers and DBs of Assembly Line, and maybe other component if any, and their interactions.

That would help in understanding the platform Assembly Line.

Assembly Line is split in different repositories: core, base, ui, etc. If it makes sense, can the diagram gather the Components coming from these different parts?

Yara Match Notification

Building on retro hunting, it would be great to add an option to keep the hunt running (for a period of time requiring renewal), which would add the user's yara rule to the Yara scanning service for incoming binaries or alternatively a batch process that runs at a regular interval. Thus if someone else submits a binary that matches another user's rule, they both get a notification of such.

Admin Notifications

It appears that announcements / notifications from CCCS are presented to all users. If you're on a Kubernetes cluster, this might be great if the updates are instantly deployed. However, if you're on docker or attempting to remain behind the latest stable build, it would seem the user's would get notification of updates to AssemblyLine that the admin hasn't actually implemented. Such notices might also be undesired for the average user, hiding other messages by the system or the admin. It might make sense to distinguish between CCCS build notifications and others. As an admin, it's great to see the changelog. I suggested filtering these messages to the admins or provide a configuration option that does so.

Bug: SHA256 Files don't retain datasource permissions

Describe the bug
Consider this scenario:

User 1 has permissions to pull samples from a restricted data sources because they're in a group that allows them to do so using a SHA256, the file is submitted into Assemblyline as normal.

User 2 does not have permissions to pull the sample from the restricted data source, and does not have the same group/context as user 1 so user 2 cannot see the submission made by user 1. However, if User 2 submits the same SHA256 in Assemblyline, because Assemblyline is not pulling the sample from external sources but from Assemblyline itself (user 1) has already submitted it, User 2 should not be able to do this, as it can be used to bypass access controls on data sources.

To Reproduce
Steps to reproduce the behaviour:

1. User 1 submits a file as TLP:R/RE
2. User 1 has access to a data source that only TLP:R/RE has access too
3. User 2 has permissions set as TLP:R/TI
4. User 2 cant see or use the submission made by User 1
5. User 2 users the SHA256 of a sample that is not available to them and submit it to AL
6. Because AL does not go to the third party source to pull that sample and analyse it the submission that was made under TLP:R/RE is now accessible under TLP:R/TI

Expected behaviour
TLP permissions should be retained even if the file is being pulled from AL itself, User 2 who does not have permissions to that external data source should not be able to use Assemblyline as a data source to get access to files they don't have the proper permissions to get access too.

Screenshots
n/a

Environment (please complete the following information if pertinent):

• Assemblyline Version:4.3.1.20
• Browser: edge

Additional context
n/a

Enhancement: Add ability to adjust a default service's scoring

Per conversation with cccs-sgaron on the Discord server.

For default services, add the capability to adjust the scores associated with heuristics while allowing that service to run normally.

Bug: extract service will enter CrashingOnLoop until Privileged mode is disabled

Describe the bug
We noticed that the extract service will enter into crashing on loop if privileged mode is enabled. This seems to be related to heartbeat, as we can see a heartbeat error when you describe the pod.

To Reproduce
Steps to reproduce the behaviour:

1. Go to Services tab
2. Click on Extract, and make sure Privileged mode is enabled
3. Save settings and wait for new pod to come up, you'll notice this pod will never come up unless you disable privileged mode

Expected behaviour
Should work in both privileged and normal mode.

Screenshots
Can replicate if required

Environment (please complete the following information if pertinent):

• Assemblyline Version: 4.3.1.16
• Browser: Edge

Additional context
Happy to provide any additional information as needed.

intezer service - new service option for disabling sub-analysis file downloads

Is your feature request related to a problem? Please describe.
For every file submitted to Intezer, all extracted files that are downloaded from that submission count against your quota.
https://support.intezer.com/hc/en-us/articles/360021366619-How-is-Your-Analysis-Quota-Calculated-

Describe the solution you'd like
Please provide a service config parameter that lets you disable the download sub-files option as this eats into the quota really quickly.

Describe alternatives you've considered
Given that a file download counts against your quota, I can't think of an alternative solution.

Additional context
Add any other context or screenshots about the feature request here.

Enhancement: Limited User Account Role

Add a limited user role, which would only be able to view its own profile and submissions. It could still be part of a classification group, so that the limited user submissions could still be viewed by an analyst assigned to the same group.
With auto-registration, this could increase the ability for users to submit samples or URLs for analysis when they should not see other submissions.

My first thought was to overload the TLP:RED classification, but I don't think that would work. It still may be appropriate to only allow the submitter to view TLP:RED (since it should be "exchanged verbally or in person"), but that is not simple either.

Scaler is too eager to scale down services with tasks

Most noticeable when core.scaler.service_defaults.min_instances: 0.

This can lead to service preemption errors on appliances or environments that want to minimize resource utilization.

Enhancement: Add Wide search to Hex Viewer

The ability to search for UTF16 strings like "http" could be simpler if there was a "wide" option.

oauth flow sign-in flow bug

Describe the bug

After updating from 4.3.0.stable62 to 4.3.1.stable23 user's are not able to sign-in to AL using oauth. The non-oauth sign-in flow still works as expected.

To Reproduce
The flow is: you get the initial sign-in page, you sign-in with your oauth provider, then it takes you back to AL where you hit sign in one more time. When you click the sign in button, it takes you to a page that says There's definitely a 'bug' here!

The message on Chrome is: Cannot read properties of undefined (reading 'toUpperCase').
The message on firefox is: e[0] is undefined.

Environment (please complete the following information if pertinent):

• Assemblyline Version: 4.3.1.stable23

Enhancement: service aliases to workaround case-sensetive ServiceNames

Possibility to add Service Aliases for the API, so we can call for alternate names/casings in addition to the real name as the API is case sensetive.

Context:

ELJeffe: When submitting a task via the api, we have "services": {"selected": ["MetaDefender", "...", "..."]}. Seems the services are case sensitive (at least on my initial tests). Would be nice if they weren't as instead of passing various tool names, I now have to specify if "metadefender" then "MetaDefender" for any particular service. I expect the same is true of others, like VirusTotal.

cccs-sgaron: It is case sensitive, you can have two service with the same name with different cases...
cccs-sgaron: Was that a good design choice, probably not, but that prevents us from removing the case sensitiveness in the API

markus-lassfolk: Would it maybe be possible for you to add Service Aliases in the API so we can call for alternate names/casings in addition to the real name?

cccs-sgaron: It's possible. Create a ticket in the Assemblyline ui repo. This might not make it as an high priority item though.

Safelist: NSRL RDSv3 Support

Describe the bug
NSRL has changed their format in RDSv3, original RDSv2 links don't work and require conversion or adoption of new format.
See: https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl/nsrl-download/current-rds

Feature Request: Blacklist services from running if submitted file is at certain TLP

If a user is submitting a file at TLP:R or TLP:R//[context] we should be able to blacklist services from running on such submissions, even if a user has requested to run that service. This will allow us to introduce riskier services without increasing risk.

Feature Request: Post-Processing enrichment on demand

As an analyst I want to be able to extract all IOCs related to a submission in the Post-Processing webhook without making additional API calls. Tags related to a submission should be present on the webhook call, this feature would be enabled using a flag in the post processing, and if enabled, the action would enrich the submission results with all the IOCs for that submission.

Enhancement: Add link to download json report

I didn't see a link in the UI to download a submission result (json) as I might get if I used the API. Would be nice to add that as an option.

Resubmit - Ignore Cache on "Use the same parameters"

When you do a Resubmit using Use the same parameters, I think you should also add Ignore result cache. Otherwise you're not really resubmitting it. You're just pulling up the cache report of the prior submission, which you were already viewing. In the original submission, there likely wasn't a cache to ignore. So the system's state has changed and I think the resubmit needs to account for that. I think the likely user intent here would be to actually run the submission again against those services.

Yara & ConfigExtractor Service will run forever if no signatures are present

Issue

If Yara and ConfigExtractor signatures are not present in the system, the Yara and ConfigExtractor service will never terminate and submissions get stuck waiting for these two services to finish.

Expected

If no signatures are present in the system, services which require signatures wont run.

Steps to reproduce

1. Make sure Yara or ConfigExtractor is Enabled
2. Make sure Yara and ConfigExtractor has no sources in the system
3. Submit a file to Assemblyline with Yara and ConfigExtractor marked to run
4. Notice Submissions get stuck and never finish, to get it unstuck you need to disable both service.

least privs approach for scaling-manager k8s service account

The scaling-manager k8s service account has overly permissive cluster level permissions that could be narrowed to only what is necessary.

e.g. These are the permissions the clusterrole edit has.

$ kubectl describe clusterrole edit
Name:         edit
Labels:       kubernetes.io/bootstrapping=rbac-defaults
              rbac.authorization.k8s.io/aggregate-to-admin=true
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources                                    Non-Resource URLs  Resource Names  Verbs
  ---------                                    -----------------  --------------  -----
  leases.coordination.k8s.io                   []                 []              [create delete deletecollection get list patch update watch]
  configmaps                                   []                 []              [create delete deletecollection patch update get list watch]
  events                                       []                 []              [create delete deletecollection patch update get list watch]
  persistentvolumeclaims                       []                 []              [create delete deletecollection patch update get list watch]
  pods                                         []                 []              [create delete deletecollection patch update get list watch]
  replicationcontrollers/scale                 []                 []              [create delete deletecollection patch update get list watch]
  replicationcontrollers                       []                 []              [create delete deletecollection patch update get list watch]
  services                                     []                 []              [create delete deletecollection patch update get list watch]
  daemonsets.apps                              []                 []              [create delete deletecollection patch update get list watch]
  deployments.apps/scale                       []                 []              [create delete deletecollection patch update get list watch]
  deployments.apps                             []                 []              [create delete deletecollection patch update get list watch]
  replicasets.apps/scale                       []                 []              [create delete deletecollection patch update get list watch]
  replicasets.apps                             []                 []              [create delete deletecollection patch update get list watch]
  statefulsets.apps/scale                      []                 []              [create delete deletecollection patch update get list watch]
  statefulsets.apps                            []                 []              [create delete deletecollection patch update get list watch]
  horizontalpodautoscalers.autoscaling         []                 []              [create delete deletecollection patch update get list watch]
  cronjobs.batch                               []                 []              [create delete deletecollection patch update get list watch]
  jobs.batch                                   []                 []              [create delete deletecollection patch update get list watch]
  daemonsets.extensions                        []                 []              [create delete deletecollection patch update get list watch]
  deployments.extensions/scale                 []                 []              [create delete deletecollection patch update get list watch]
  deployments.extensions                       []                 []              [create delete deletecollection patch update get list watch]
  ingresses.extensions                         []                 []              [create delete deletecollection patch update get list watch]
  networkpolicies.extensions                   []                 []              [create delete deletecollection patch update get list watch]
  replicasets.extensions/scale                 []                 []              [create delete deletecollection patch update get list watch]
  replicasets.extensions                       []                 []              [create delete deletecollection patch update get list watch]
  replicationcontrollers.extensions/scale      []                 []              [create delete deletecollection patch update get list watch]
  ingresses.networking.k8s.io                  []                 []              [create delete deletecollection patch update get list watch]
  networkpolicies.networking.k8s.io            []                 []              [create delete deletecollection patch update get list watch]
  poddisruptionbudgets.policy                  []                 []              [create delete deletecollection patch update get list watch]
  deployments.apps/rollback                    []                 []              [create delete deletecollection patch update]
  deployments.extensions/rollback              []                 []              [create delete deletecollection patch update]
  pods/eviction                                []                 []              [create]
  serviceaccounts/token                        []                 []              [create]
  pods/attach                                  []                 []              [get list watch create delete deletecollection patch update]
  pods/exec                                    []                 []              [get list watch create delete deletecollection patch update]
  pods/portforward                             []                 []              [get list watch create delete deletecollection patch update]
  pods/proxy                                   []                 []              [get list watch create delete deletecollection patch update]
  secrets                                      []                 []              [get list watch create delete deletecollection patch update]
  services/proxy                               []                 []              [get list watch create delete deletecollection patch update]
  bindings                                     []                 []              [get list watch]
  endpoints                                    []                 []              [get list watch]
  limitranges                                  []                 []              [get list watch]
  namespaces/status                            []                 []              [get list watch]
  namespaces                                   []                 []              [get list watch]
  persistentvolumeclaims/status                []                 []              [get list watch]
  pods/log                                     []                 []              [get list watch]
  pods/status                                  []                 []              [get list watch]
  replicationcontrollers/status                []                 []              [get list watch]
  resourcequotas/status                        []                 []              [get list watch]
  resourcequotas                               []                 []              [get list watch]
  services/status                              []                 []              [get list watch]
  controllerrevisions.apps                     []                 []              [get list watch]
  daemonsets.apps/status                       []                 []              [get list watch]
  deployments.apps/status                      []                 []              [get list watch]
  replicasets.apps/status                      []                 []              [get list watch]
  statefulsets.apps/status                     []                 []              [get list watch]
  horizontalpodautoscalers.autoscaling/status  []                 []              [get list watch]
  cronjobs.batch/status                        []                 []              [get list watch]
  jobs.batch/status                            []                 []              [get list watch]
  endpointslices.discovery.k8s.io              []                 []              [get list watch]
  daemonsets.extensions/status                 []                 []              [get list watch]
  deployments.extensions/status                []                 []              [get list watch]
  ingresses.extensions/status                  []                 []              [get list watch]
  replicasets.extensions/status                []                 []              [get list watch]
  nodes.metrics.k8s.io                         []                 []              [get list watch]
  pods.metrics.k8s.io                          []                 []              [get list watch]
  ingresses.networking.k8s.io/status           []                 []              [get list watch]
  poddisruptionbudgets.policy/status           []                 []              [get list watch]
  serviceaccounts                              []                 []              [impersonate create delete deletecollection patch update get list watch]

I did some analysis on this a while ago and narrowed the permissions to only what is needed. I confirmed on an EKS deployment over multiple months and more recently on a microk8s cluster with RBAC enabled.

Please consider this PR: CybercentreCanada/assemblyline-helm-chart#63

Typing affecting tests for services

Ref: CybercentreCanada/assemblyline-service-sigma#40 (review)

https://github.com/CybercentreCanada/assemblyline-base/blob/c560c7c2a77c6c50a2cd77ebac6bd5324ef0b713/assemblyline/odm/models/result.py#L140

XLSB Processing Enhancements

Full XLSB code deobfuscation is probably out of scope for this service, but extracting and/or combining related subfiles for submission to another service may be appropriate.

https://isc.sans.edu/forums/diary/XLSB+Files+Because+Binary+is+Stealthier+Than+XML/28476/

https://www.virustotal.com/gui/file/156796008e389996f37f423c3f4d0be316a1b7effe14fbe09e2bb242c92bd3ef
https://www.virustotal.com/gui/file/5a6157eefc8d0b1089a5bfdee351379b27baff4c40b432fd22e0cbe1f6102fab

Fetchers need an optional proxy

None of the fetchers (Harbor, Docker, DockerHub) are setup to use a proxy. When trying to add a new service that includes $SERVICE_TAG, I get the below error.

https://github.com/CybercentreCanada/assemblyline-core/blob/33edc58aa50032e4f8f9845247cd495309d03b7c/assemblyline_core/updater/helper.py#L184

File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/flask/app.py", line 2525, in wsgi_app response = self.full_dispatch_request() 
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/flask/app.py", line 1822, in full_dispatch_request rv = self.handle_user_exception(e) 
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/flask/app.py", line 1820, in full_dispatch_request rv = self.dispatch_request() 
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/flask/app.py", line 1796, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) 
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline_ui/api/base.py", line 189, in base return func(*args, **kwargs) 
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline_ui/api/v4/service.py", line 256, in add_service _, tag_name, _ = get_latest_tag_for_service(tmp_service, config, LOGGER) 
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline_core/updater/helper.py", line 138, in get_latest_tag_for_service tags = _get_dockerhub_tags(image_name, update_channel) 
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/assemblyline_core/updater/helper.py", line 184, in _get_dockerhub_tags resp = requests.get(url) 
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) 
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) 
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/requests/sessions.py", line 587, in request resp = self.send(prep, **send_kwargs) 
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/requests/sessions.py", line 701, in send r = adapter.send(request, **kwargs) 
File "/var/lib/assemblyline/.local/lib/python3.9/site-packages/requests/adapters.py", line 565, in send raise ConnectionError(e, request=request) 
ConnectionError: HTTPSConnectionPool(host='registry.hub.docker.com', port=443): Max retries exceeded with url: /v2/repositories/cccs/assemblyline-service-avclass/tags?page_size=5&page=1&name=stable (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fc9cfa3bcd0>: Failed to establish a new connection: [Errno 111] Connection refused'))

SAML Authentication Support

Would like to request SAML Authentication support. I'm currently using LDAP in our dev, but production will need to be able to use SAML to authenticate users. Wanted to try and get this on the roadmap. Thanks

Feature Request: auto_properties should accept array when adding and removing roles

Is your feature request related to a problem? Please describe.
When setting users to "custom" and manually adding roles back in based on groups, using the "type: role" we should be able to specify an array of values instead of copy pasting the field and changing the role value, this should also be an option for removing roles. As it stands if we want to add or remove roles we need to create a new field group with patterns, type and value, as it only accepts a keyword value.

eg:

            - field: groups
              pattern: ^pattern1$
              type: role
              value: alert_manage

            - field: groups
              pattern: ^pattern1$
              type: role
              value: alert_view

            - field: groups
              pattern: ^pattern1$
              type: role
              value: apikey_access

            - field: groups
              pattern: ^pattern1$
              type: role
              value: file_detail

Describe the solution you'd like

            - field: groups
              pattern: ^pattern1$
              type: role
              value: [file_detail, apikey_access, alert_view, alert_manage]

Describe alternatives you've considered

When trying the above exceptions is thrown;

raise ValueError(f"[{self.name or self.parent_name}] {value} not in the possible values: { self.values}")\nValueError: [roles] ['alert_manage', 'alert_view', 'apikey_access', 'file_detail', 'heuristic_view', 'obo_access', 'replay_trigger', 'safelist_view', 'safelist_manage', 'signature_view', 'signature_download', 'submission_create', 'submission_delete', 'submission_manage', 'submission_view', 'workflow_manage', 'workflow_view', 'replay_system', 'archive_view', 'archive_manage', 'archive_trigger', 'self_manage'] not in the possible values: {'bundle_download', 'signature_ import', 'signature_download', 'file_download', 'submission_view', 'self_manage', 'alert_manage', 'signature_manage', 'administration', 'archive_download', 'replay_system', 'replay_trigger', 'alert_view', 'apikey_access', 'workflow_manage', 'heuristic_view', 'submission_manage', 'safelist_view', 'archive_view', 'workflow_view', 'submission_delete', 'submission_create', 'safelist_manage', 'signature_view', 'archive_trigger', 'obo_access', 'file_detail', 'archive_manage'}\n"}

The same is true if we try:

            - field: groups
              pattern: ^pattern1$
              type: role
              value: file_detail, apikey_access, alert_view, alert_manage

or

            - field: groups
              pattern: ^pattern1$
              type: role
              value: ["file_detail", "apikey_access", "alert_view", "alert_manage"]

or

            - field: groups
              pattern: ^pattern1$
              type: role
              value: 
                   -  "file_detail"
                   -  "apikey_access"
                   -  "alert_view"
                   - "alert_manage"

Additional context
unless I'm doing something wrong here I think collections of roles are not allowed.

Similar issue on the documentation as seen below:

Retro Hunting

My user's most requested feature for AssemblyLine is the ability to do retro hunting. We would like the ability for a user to be able to submit a yara rule that scans our filestore.

EmlParser Header Filter Fails on MSG

The EmlParser does not perform case insensitive filtering of email headers for submissions with the type "document/office/email".

To Reproduce
Steps to reproduce the behavior:

1. Add "DKIM-Signature" to header_filter in the EmlParser service
2. Submit MSG file that contains a DKIM-Signature header
3. View EmlParser results in the submission
4. DKIM Signature is included in the Email Headers section for the EmlParser

Expected behavior
The header_filter should be consistent across submission file types

Additional context
The msg parser does not make case-insensitive comparisons
https://github.com/CybercentreCanada/assemblyline-service-emlparser/blob/d1035125f6222255c2e7846dd3d1ef214e1947f1/emlparser/emlparser.py#L63

The eml parser makes case-insensitive comparisons:
https://github.com/CybercentreCanada/assemblyline-service-emlparser/blob/d1035125f6222255c2e7846dd3d1ef214e1947f1/emlparser/emlparser.py#L412