thehive-project/cortex4py

Python API Client for Cortex

cortex4py's Issues

Add support to responders

The goal of this feature is to add a controller that handles all the operations that can be executed agains a responder

Can't get jobs details

I'm trying to write a little tool to keep monitored range limits.
By the way, I can't get any kind of information.

My main code is:

    for analyzer in analyzer_list:
        query = And(Eq('analyzer', analyzer["analyzer"]))
        jobs = api.jobs.find_all(query)

        total_jobs = jobs.__len__()
        r = {
            "analyzer": analyzer["analyzer"],
            "total_jobs_performed_today": str(total_jobs)
        }

Any suggestions?

The Hive 4.1.24-1 and Cortex 3.1.7-1 Auth_Error

Get analyser jobs per case

Trying to create a reporting responder using the following reporter as a base. I want the responder to fetch all analyzer job results for each observable in case. The purpose is to have all information included in the report(for a case) so the analyst can edit/remove what is required or not in the final report. I've checked the Cortex4py documentation and you can get (ex: per name or id)the analyzer and jobs details but there is no way to narrow it down to a specific case(using the case ID for example). Basically, I would like to be able to programatically get all analyzer job results for each observable in a case.

Is it something that is feasible? If not, is it something that could be implemented?

Thanks

Bug in get_analyzers function

The code of the function:

api.py in get_analyzers(self, data_type)
    153         )
    154         if data_type is not None:
--> 155             return self.analyzers.find_all()
    156         else:
    157             return self.analyzers.get_by_type(data_type)

the line 154 shouldn't contain not.

[BUG] SERVICE UNAVAILABLE - Cortex service is unavailable, is configuration correct ?

We are trying to configure our Cortex instance in your app so Splunk can talk to it, howerver we are seeing following error when trying to List Cortex jobs form within Splunk:

ERROR cortex:112 - [C26-ERROR] SERVICE UNAVAILABLE - Cortex service is unavailable, is configuration correct ?

Steps taken to troubleshoot

1. Installed fresh TheHive-Cortex app on our testing SH
2. Created account used for Cortex (Settings->Account) using Cortex API token as password
3. Created instance with default values as shown in screenshot below
4. Tried to list Cortex jobs (Cortex->Cortex Jobs -> LIST)

Splunk log

2021-12-03 12:08:10,772 DEBUG	common:31 - [S2] Logging mode set to DEBUG
<KV Stored content redacted>
2021-12-03 12:08:11,090 DEBUG	common:128 - [S25] Successfully recovering passwords from storage passwords
2021-12-03 12:08:11,090 DEBUG	cortex:57 - [C6] Settings recovered
2021-12-03 12:08:11,090 DEBUG	common:217 - [S55] Getting this parameter: 100
2021-12-03 12:08:11,091 DEBUG	common:223 - [S60] Getting this parameter: -createdAt
2021-12-03 12:08:11,091 DEBUG	common:177 - [S35] This instance ID (fa082ff0) returns: apiuser
2021-12-03 12:08:11,091 DEBUG	common:163 - [S30] This instance ID (fa082ff0) returns: {'account_name': 'Cortex', 'authentication_type': 'api_key', 'client_cert': '-', 'host': '127.0.0.1', 'organisation': None, 'port': 9001, 'proxy_account': '-', 'proxy_url': '-', 'type': 'Cortex3', 'uri': '/', 'verify': False, '_user': 'nobody', '_key': '61aa075809cf067420127851', 'proxies': None, 'username': 'apiuser', 'password': '**********'}
2021-12-03 12:08:11,091 DEBUG	common:185 - [S40] this instance id (fa082ff0) returns: authentication_type=api_key
2021-12-03 12:08:11,091 DEBUG	common:185 - [S40] this instance id (fa082ff0) returns: proxies=None
2021-12-03 12:08:11,092 DEBUG	common:185 - [S40] this instance id (fa082ff0) returns: client_cert=-
2021-12-03 12:08:11,092 DEBUG	common:185 - [S40] this instance id (fa082ff0) returns: verify=False
2021-12-03 12:08:11,092 DEBUG	common:185 - [S40] this instance id (fa082ff0) returns: organisation=None
2021-12-03 12:08:11,092 DEBUG	common:185 - [S40] this instance id (fa082ff0) returns: type=Cortex3
2021-12-03 12:08:11,092 DEBUG	cortex:79 - [C8] Cortex instance will be initialized with an API Key (not a password)
2021-12-03 12:08:11,093 DEBUG	cortex:102 - [C20] Cortex object instanciated
2021-12-03 12:08:11,115 ERROR	cortex:112 - [C26-ERROR] SERVICE UNAVAILABLE - Cortex service is unavailable, is configuration correct ?

To avoid any problems (SSL, firewall, etc) we've tried to forward the port from Cortex instance to the Splunk SH machine so Cortex actually listens on localhost (127.0.0.1) and default port (9001), so following command works:

curl -H "Authorization: Bearer [REDACTED]" http://127.0.0.1:9001/api/analyzer

Cortex Version: 3.1.1-1
Splunk version: 8.2.2.1
TheHive-Cortex version: 2.1.5

Small naming issues in Cortex4py usage.md examples

Just to note that I have noticed two small typing errors while going through Cortex4py documentation.

1. In the example showing how to manipulate organizations as orgadmin change 'status','Active' to 'status','Ok':

From

users = api.organizations.get_users(org.id, Eq('status', 'Active'), range='0-5', sort='-createdAt')

To

users = api.organizations.get_users(org.id, Eq('status', 'Ok'), range='0-5', sort='-createdAt')

Otherwise active users won't be shown.

2. In the example showing how to manipulate analyzers, to enable an analyzer change
   Change the "api_key" parameter in configuration to "key":

From

analyzer = api.analyzers.enable('Test_1_0', {
  "configuration": {
    "api_key": "XXXXXXXXXXXXXx",
    "proxy_http": "http://localhost:9999",
    "proxy_https": "http://localhost:9999",
    "auto_extract_artifacts": False,
    "check_tlp": True,
    "max_tlp": 2
  }

To

analyzer = api.analyzers.enable('Test_1_0', {
  "configuration": {
    "key": "XXXXXXXXXXXXXx",
    "proxy_http": "http://localhost:9999",
    "proxy_https": "http://localhost:9999",
    "auto_extract_artifacts": False,
    "check_tlp": True,
    "max_tlp": 2
  }

Otherwise "Invalid input Excetion" will be returned.

Nothing big at all, just small little typing errors which might not necessarily be easy to notice for everyone as they took me quite some time to figure out! Hope this helps :)

AttributeError NoneType exception from run_by_name if no analyzers are installed

In a testing environment I have a Cortex in a Docker container that is still missing analysers. So the list of analysers returned by the API is empty. This causes the following exception when calling cortex4py.analyzers.run_by_name() with any name:

'NoneType' object has no attribute 'id'
Traceback (most recent call last):
  File "/home/m/PeekabooAV-Installer/PeekabooAV/peekaboo/ruleset/engine.py", line 193, in run
    result = rule.evaluate(sample)
  File "/home/m/PeekabooAV-Installer/PeekabooAV/peekaboo/ruleset/rules.py", line 754, in evaluate
    self.submit_to_cortex(sample, cortex_analyzer)
  File "/home/m/PeekabooAV-Installer/PeekabooAV/peekaboo/ruleset/rules.py", line 226, in submit_to_cortex
    job_id = self.cortex.submit(sample, analyzer)
  File "/home/m/PeekabooAV-Installer/PeekabooAV/peekaboo/toolbox/cortex.py", line 543, in submit
    job = self.api.analyzers.run_by_name(analyzer.name, params)
  File "/home/m/pd/lib/python3.8/site-packages/cortex4py/controllers/analyzers.py", line 89, in run_by_name
    return self.run_by_id(analyzer.id, observable, **kwargs)
AttributeError: 'NoneType' object has no attribute 'id'

I can handle the NoneType exception in my code but wanted to report the issue here in case you think it should rather be wrapped into a module-specific exception such as cortex4py.exceptions.NotFoundError or cortex4py.analyzers.run_by_name() signal error by e.g. returning None.

Cortex Authentication

Are those samples still valid?
When i try to connect to cortex api with a sample, cortex API response is that Authentication is invalid. I did not pass the api key. The function does not accept such parameter.

Wait for the job to finish

I is it way to wait for the job to finish like in this example :

job1 = api.analyzers.run_by_name('FileInfo_6_0', {
    'data': '/bin/ls',
    'dataType': 'file'
}, async=True)

# The next instruction should return only Success or Failure, because we reached it after the job finished executing.
print(job1.status)

Thank you in advance for your response.

Submitting the PAP level of data is not possible

Since instead of passing the content of observable as post, the post dictionary is recreated only with the dataType and tlp level, it is currently not possible to submit the PAP level to the job:

Cortex4py/cortex4py/controllers/analyzers.py

Lines 49 to 56 in 629069a

	def run_by_id(self, analyzer_id, observable, **kwargs) -> Job:
	tlp = observable.get('tlp', 2)
	data_type = observable.get('dataType', None)
	post = {
	'dataType': data_type,
	'tlp': tlp
	}

analyzer.run_by_id fails at least 30% of the time

There is a huge difference in quality performance between using cortex4py and cortex API.

When I use cortex API to run analizer job with post requests I get valid response every time.
When I use cortex4py to do the same - the script fails at very high rate.

Consider two functions:

1. Using cortex4py:

def run_vt(domain):
    result = None
    job = capi.analyzers.run_by_name('VirusTotal_GetReport_3_0', {
        'data': str(domain),
        'dataType': 'domain',
        'tlp': 1,
        'message': 'alerts verificiation'
    }, force=1)
    try:
        result = job.json()
        return result
    
    except Exception as ex:
        print(get_time_now(), status('EXPT'), 'Exception running VT analizer:', ex)
        return result
    
    return result

2. Using Cortex API directly:

def run_vt_request(domain):
    headers = {}
    headers.update(cortex_auth)
    headers.update(content_type)

    url = cortex_url + '/api/analyzer/ [id] /run'

    j = {
        'data': domain,
        'dataType': 'domain',
        'tlp': 1
    }
    resp = None
    
    try:    
        resp = requests.post(url, headers=headers, json=j, verify=False)
        return resp.json()
    except Exception as ex:
        print(get_time_now(), status('EXPT'), 'Exception running VT analizer:', ex)
        return resp
    return resp

First one fails a lot (HTTP Error 500, Invalid input exception).
Second did not failed once.

I suspect force parameter placement or interpretation in the cortex4py implementation might be at fault. I am not completely sure.

Specifics of job.status and report['success']

Hi,

Can you give some clarification to the differences between job.status and report['success'] fields, please? In particular when job.status would indicate success but the report indicates unsuccessful.

TIA!

Invalid input exception when calling api.jobs.get_report_async

Following exception is raised when calling api.jobs.get_report_async immediately after starting job with api.analyzers.run_by_name.

cortex4py.exceptions.InvalidInputError: Invalid input exception

> requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://<IP>/api/job/<job_id>/waitreport?atMost=Inf

If I insert a five-second sleep statement before calling get_report_async then it succeeds. The same error occurs if I request the waitreport API endpoint with the requests module with the atMost=Inf parameter.

If I remove the atMost=Inf parameter the request succeeds without needing the sleep statement (i.e. it waits for the analyzer to complete).

Looking at the cortex logs, the following exception is thrown for the request:

2019-04-12 15:00:27,364 [INFO] from org.thp.cortex.services.ErrorHandler in application-akka.actor.default-dispatcher-166 - GET /api/job/<job_id>/waitreport?atMost=Inf returned 500
java.lang.ClassCastException: scala.concurrent.duration.Duration$$anon$2 cannot be cast to scala.concurrent.duration.FiniteDuration
        at org.thp.cortex.controllers.JobCtrl.$anonfun$waitReport$2(JobCtrl.scala:148)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:303)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 

Add support to files as output of analyzers

Analyzers are not able to produce files as analysis output before Cortex 3. This issue aims to prepare the availability of this feature.

Get quota used of analyzer rates

Hi, any way to get the used quota of an analyzer?
Thanks

Document how to use the query function

Is someone able to send me in the right direction to learn more about how the query object is used to get a subset of analyzers, please?

My use case is getting a list of analyzers for dataType ip, domain etc. with a minimum or maximum tlp value. At the moment I have the following which gets analyzers of ip type:

    query = In('dataTypeList', ['ip'])

I tried something like the following to evaluate the max_tlp value within the configuration object, but that failed (I didn't expect it would work!):

    query = And(In('dataTypeList', ['ip']), Child('configuration', 'max_tlp==2'))

So how would I access and evaluate fields within the configuration fields of an analyzer?

TIA!