alienvault-otx/otx-python-sdk

The Python SDK for AlienVault OTX

otx-python-sdk's Issues

Sorting of response

Hello Everyone,

I've noticed that the default sorting of the API responses are ascending.
This of course makes searching for recent Pulses a pain, I would have to pull huge amounts of data and sort locally, which is an unnecessary strain on the API
Would anyone be interested in adding a parameter to sort descending?

Regards,

Linus

Otx

Search queries are not URL encoded

If I try to run a search against /api/v1/search/pulses with spaces in the query, I get a Bad Request:

>>> otx.search_pulses('phpMyAdmin honeypot logs for 2016-10-01')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "OTXv2.py", line 293, in search_pulses
    return self._get_paginated_resource(search_pulses_url, max_results=max_results)
  File "OTXv2.py", line 317, in _get_paginated_resource
    json_data = self.get(next_page_url)
  File "OTXv2.py", line 79, in get
    raise BadRequest("Bad Request")

OTXv2.BadRequest: 'Bad Request'

This is because the query in the search_pulses method is not URL encoded, so it tries to GET something like this: https://otx.alienvault.com/api/v1/search/pulses?q=phpMyAdmin honeypot logs for 2016-10-01&.

To fix this we can use urllib.quote_plus for Python 2 (or urllib.parse.quote_plus() for Python 3) to properly encode our query.

>>> import urllib
>>> urllib.quote_plus('phpMyAdmin honeypot logs for 2016-10-01')
'phpMyAdmin+honeypot+logs+for+2016-10-01'

So a quick fix would be to update the q=query in the search_pulses method to q=urllib.quote_plus(q), like so:

def search_pulses(self, query, max_results=25):
        search_pulses_url = self.create_url(SEARCH_PULSES, q=urllib.quote_plus(q), page=1, limit=20)
        return self._get_paginated_resource(search_pulses_url, max_results=max_results)

I'm not sure if this is the norm in APIs, but wouldn't it make sense to update your python API module to do the URL encoding? If you're expecting the user to do it, then maybe mention that under the API module documentation?

Incomplete parsing of plugins field in results

Only few plugins are being parsed in the result that's being returned in
result = otx.get_indicator_details_full(hash_type, hash)
Fields like msdefender, exiftool and some others aren't being taken into consideration in the def file(otx, hash) function. This is leading to some hashes not being recognised as malicious even though they have been listed as malicious on OTX site

Python not book does not show where to set the API key

The python notebook says to replace YOUR_KEY with your actual API key. It doesnt say anywhere where to specify the key.

otx.getall() having error "Exceeded maximum number of retries"

---

MaxRetryError Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
448 retries=self.max_retries,
--> 449 timeout=timeout
450 )

C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
730 release_conn=release_conn,
--> 731 body_pos=body_pos, **response_kw)
732

C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
730 release_conn=release_conn,
--> 731 body_pos=body_pos, **response_kw)
732

C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
730 release_conn=release_conn,
--> 731 body_pos=body_pos, **response_kw)
732

C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
730 release_conn=release_conn,
--> 731 body_pos=body_pos, **response_kw)
732

C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
730 release_conn=release_conn,
--> 731 body_pos=body_pos, **response_kw)
732

C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
710 try:
--> 711 retries = retries.increment(method, url, response=response, _pool=self)
712 except MaxRetryError:

C:\ProgramData\Anaconda3\lib\site-packages\urllib3\util\retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
398 if new_retry.is_exhausted():
--> 399 raise MaxRetryError(_pool, url, error or ResponseError(cause))
400

MaxRetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20 (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

RetryError Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in get(self, url, **kwargs)
175 headers=self.headers,
--> 176 proxies=self.proxies,
177 )

C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py in get(self, url, **kwargs)
545 kwargs.setdefault('allow_redirects', True)
--> 546 return self.request('GET', url, **kwargs)
547

C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
532 send_kwargs.update(settings)
--> 533 resp = self.send(prep, **send_kwargs)
534

C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py in send(self, request, **kwargs)
645 # Send the request
--> 646 r = adapter.send(request, **kwargs)
647

C:\ProgramData\Anaconda3\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
506 if isinstance(e.reason, ResponseError):
--> 507 raise RetryError(e, request=request)
508

RetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20 (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

RetryError Traceback (most recent call last)
in
----> 1 pulses = otx.getall()

C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in getall(self, modified_since, author_name, limit, max_page, max_items, iter)
393 return self.walkapi(
394 self.create_url(SUBSCRIBED, **args), iter=iter,
--> 395 max_page=max_page, max_items=max_items
396 )
397

C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in walkapi(self, url, iter, max_page, max_items, method, body)
372 return self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body)
373 else:
--> 374 return list(self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body))
375
376 def getall(self, modified_since=None, author_name=None, limit=20, max_page=None, max_items=None, iter=False):

C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in walkapi_iter(self, url, max_page, max_items, method, body)
353
354 if method == 'GET':
--> 355 data = self.get(next_page_url)
356 elif method == 'POST':
357 data = self.post(next_page_url, body=body)

C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in get(self, url, **kwargs)
178 return self.handle_response_errors(response).json()
179 except requests.exceptions.RetryError:
--> 180 raise RetryError()
181
182 def patch(self, url, body, **kwargs):

RetryError: 'Exceeded maximum number of retries'

is_malicious.py "ResourceWarning"

Hello there,

I would like to report a Python warning:

$ python3 -Wall is_malicious.py -file /etc/passwd
/home/test/is_malicious.py:63: ResourceWarning: unclosed file <_io.BufferedReader name='/etc/passwd'>
  hash = hashlib.md5(open(args['file'], 'rb').read()).hexdigest()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Unknown or not identified as malicious

Cheers!

Write tests and run them on Travis CI

Thoughts?

"Did not find expected key"

Configured per instructions but receiving the following in startup log.

`[1378] 27/6/2017 -- 13:10:58 - (conf-yaml-loader.c:241) (ConfYamlParse) -- Including configuration file /etc/suricata/otx-suricata.yaml.
[1378] 27/6/2017 -- 13:10:58 - (conf-yaml-loader.c:265) (ConfYamlParse) -- Configuration node 'rule-files' redefined.
[1378] 27/6/2017 -- 13:10:58 - (conf-yaml-loader.c:180) (ConfYamlParse) -- [ERRCODE: SC_ERR_CONF_YAML_ERROR(242)] - Failed to parse configuration file at line 9: did not find expected key

[1378] 27/6/2017 -- 13:10:58 - (conf-yaml-loader.c:148) (ConfYamlHandleInclude) -- [ERRCODE: SC_ERR_CONF_YAML_ERROR(242)] - Failed to include configuration file /etc/suricata/otx-suricata.yaml`

AttributeError: 'module' object has no attribute 'RetryError'

Receiving the following error upon first run:

`~/OTX-Suricata/otx-suricata# python test_rules.py
E

ERROR: testRuleGenerate (main.TestGenerateRules)

Traceback (most recent call last):
File "test_rules.py", line 41, in testRuleGenerate
self.suricata_client.generate_rules(True, True)
File "/root/OTX-Suricata/otx-suricata/suricata.py", line 37, in generate_rules
for pulse in self.client.getall_iter():
File "/usr/local/lib/python2.7/dist-packages/OTXv2.py", line 265, in walkapi_iter
data = self.get(next_page_url)
File "/usr/local/lib/python2.7/dist-packages/OTXv2.py", line 115, in get
except requests.exceptions.RetryError:
AttributeError: 'module' object has no attribute 'RetryError'

---

Ran 1 test in 5.509s

FAILED (errors=1)`

504 error when using otx.getall() method

We are facing the following error when making a call to the getAll() method using the otxv2 sdk.

pulses = otx.getall() File "<user_dir>/env/lib/python3.6/site-packages/OTXv2.py", line 341, in getall return self.walkapi(self.create_url(SUBSCRIBED, **args), iter=iter, max_page=max_page, max_items=max_items) File "<user_dir>/env/lib/python3.6/site-packages/OTXv2.py", line 322, in walkapi return list(self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body)) File "<user_dir>/env/lib/python3.6/site-packages/OTXv2.py", line 303, in walkapi_iter data = self.get(next_page_url) File "<user_dir>/env/lib/python3.6/site-packages/OTXv2.py", line 148, in get return self.handle_response_errors(response).json() File "<user_dir>/env/lib/python3.6/site-packages/OTXv2.py", line 131, in handle_response_errors raise Exception("Unexpected http code: %r, response=%r", response.status_code, _response_json()) {{Exception: ('Unexpected http code: %r, response=%r', 504, {'internal_error': 'Unable to decode response json: Expecting value: line 1 column 1 (char 0)'} )}}

Version of OTXV2 we are using - 1.5.2

Please let me know if any other information is needed, thanks.

Add usage commands & initial output

Wrong repo, ignore.

AttributeError: 'NoneType' object has no attribute 'read'

When looping through IP's using the otx.get_indicator_details_full function, I get an error - AttributeError: 'NoneType' object has no attribute 'read'.

Traceback (most recent call last):

  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 375, in trace_task

    R = retval = fun(*args, **kwargs)

  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 632, in __protected_call__

    return self.run(*args, **kwargs)

  File "/usr/src/app/app/tasks.py", line 32, in process_ips

    document = IPHandler().process(ip)

  File "/usr/src/app/app/engine/ip_handler.py", line 39, in process

    OTX_response = self.get_OTX_data(ip)

  File "/usr/src/app/app/engine/ip_handler.py", line 15, in get_OTX_data

    otx_data = self.otx.get_indicator_details_full(IndicatorTypes.IPv4, ip)

  File "/usr/local/lib/python3.6/site-packages/OTXv2.py", line 492, in get_indicator_details_full

    indicator_dict[section] = self.get(indicator_url)

  File "/usr/local/lib/python3.6/site-packages/OTXv2.py", line 83, in get

    data = response.read().decode('utf-8')

AttributeError: 'NoneType' object has no attribute 'read' 

Using Python Version - 3.6.4

List index out of range from mtime

>>> mtime = (datetime.now() - timedelta(days=1)).isoformat()
>>> mtime
'2015-12-09T14:33:11.722751'
>>> events = otx.getevents_since(mtime)
>>> json_normalize(events)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/pandas/io/json.py", line 684, in json_normalize
    if any([isinstance(x, dict) for x in compat.itervalues(data[0])]):
IndexError: list index out of range

body parameter documentation for edit_pulse() should indicate the it's a delta set, not a complete set.

Line 411, the documentation for the body parameter for edit_pulse(), should be changed to indicate it's actual the delta set, or set of adds and removes, rather than a "complete set".

Suspect the currently wording was not changed as a result of copying from what is now line 426, the new_indicators parameter documentation for replace_pulse_indicators.

Error while pulling pulses

When i am trying to fetch the pulses. It is throwing an "OSError: Tunnel connection failed: 407 Proxy Authentication Required" error. Is it because i am using office network. If yes, than how to resolve the issue.

Provide a documentation of functions for the OTXv2.py file

Hello,

Could you provide a documentation for OTXv2.py file ?

I know it is not a complex file and reading the source provide the information, but it would be more convenient to provide documentation and to be able to refer to, for instance a readthedocs.io page.

Cheers!

Allow requests library to take verify and cert parameters

For improved proxy support, could we add the verify and cert parameters to the requests query? I have a working copy and can do a pull request to add this functionality.

Error when requesting a non-existent pulse

When requesting an non-existent pulse, with pip version 1.2:

indicators = otx.get_pulse_indicators("aaaaaaaaaa")
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-6-78f47ae1d67a> in <module>()
----> 1 indicators = otx.get_pulse_indicators("aaaaaaaaaa")

/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in get_pulse_indicators(self, pulse_id, limit)
    412         next_page_url = self.create_url(PULSE_DETAILS + str(pulse_id) + "/indicators", limit=limit)
    413         while next_page_url:
--> 414             json_data = self.get(next_page_url)
    415             for r in json_data["results"]:
    416                 indicators.append(r)

/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in get(self, url)
     81             else:
     82                 raise e
---> 83         data = response.read().decode('utf-8')
     84         json_data = json.loads(data)
     85         return json_data

AttributeError: 'NoneType' object has no attribute 'read'

With masterbranch (commit da9cd34 ):

indicators = otx.get_pulse_indicators("aaaaaaaaaa")
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-3-78f47ae1d67a> in <module>()
----> 1 indicators = otx.get_pulse_indicators("aaaaaaaaaa")

/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in get_pulse_indicators(self, pulse_id, limit)
    402         :return: Indicator list
    403         """
--> 404         return self.walkapi(self.create_url(PULSE_DETAILS + str(pulse_id) + "/indicators", limit=limit))
    405 
    406 

/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in walkapi(self, url, iter, max_page)
    272             return self.walkapi_iter(url, max_page=max_page)
    273         else:
--> 274             return list(self.walkapi_iter(url, max_page=max_page))
    275 
    276     def getall(self, modified_since=None, author_name=None, limit=20, max_page=None, iter=False):

/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in walkapi_iter(self, url, max_page)
    263                 break
    264 
--> 265             data = self.get(next_page_url)
    266             for el in data['results']:
    267                 yield el

/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in get(self, url, **kwargs)
    112                 proxies=self.proxies,
    113             )
--> 114             return self.handle_response_errors(response).json()
    115         except requests.exceptions.RetryError:
    116             raise RetryError()

/home/user/envs/analysis3/lib/python3.5/site-packages/OTXv2.py in handle_response_errors(cls, response)
     95             raise BadRequest(_response_json())
     96         elif str(response.status_code)[0] != "2":
---> 97             raise Exception("Unexpected http code: %r, response=%r", response.status_code, _response_json())
     98 
     99         return response

Exception: ('Unexpected http code: %r, response=%r', 404, {'internal_error': 'Unable to decode response json: Expecting value: line 1 column 1 (char 0)'})

It would be way better to raise a custom exception here.

Throttling During High Volume Ingestion

Hello,

I am currently creating a pool of processes that simultaneously hit the API with multiple IOC's to retrieve indicator details.

The API stops pulling after about 670 MB of IOC data.

I made sure my machine is hardwired to the internet and the network has not gone down during my runs.

I have also made sure my machine doesn't go to sleep or that anything is stopping the script from running.

I am handling failing IOC's using exceptions and they get logged with no issues.

After pulling details for about 3-4k IOC's the API hangs.

Any idea what could be causing this issue?

Thank you,
Baalgar

API_V1_ROOT missing backslash causing requests to fail

API_V1_ROOT is "api/v1" when it looks like it should be "/api/v1" or requests to the API fail because otx.alienvault.comapi is not a valid hostname for HTTP requests. I'm on Python 2.7.11 just fyi.

• Line: https://github.com/AlienVault-Labs/OTX-Python-SDK/blob/master/OTXv2.py#L17

Simply prepending a backslash to the API_V1_ROOT variable fixed this for me when I reinstalled the latest master branch locally. Added the backslash to the line above, ran pip install . --upgrade, and it worked as expected.

API call & Traceback:

>>> mirai_iocs = otx.get_pulse_indicators('58e39804b789b25b15083fc6', limit=219)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    mirai_iocs = otx.get_pulse_indicators('58e39804b789b25b15083fc6', limit=219)
  File "/usr/local/lib/python2.7/site-packages/OTXv2.py", line 404, in get_pulse_indicators
    return self.walkapi(self.create_url(PULSE_DETAILS + str(pulse_id) + "/indicators", limit=limit))
  File "/usr/local/lib/python2.7/site-packages/OTXv2.py", line 274, in walkapi
    return list(self.walkapi_iter(url, max_page=max_page))
  File "/usr/local/lib/python2.7/site-packages/OTXv2.py", line 265, in walkapi_iter
    data = self.get(next_page_url)
  File "/usr/local/lib/python2.7/site-packages/OTXv2.py", line 112, in get
    proxies=self.proxies,
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 501, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 487, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='otx.alienvault.comapi', port=443): Max retries exceeded with url: /v1/pulses/58e39804b789b25b15083fc6/indicators?limit=219 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x10c0d5110>: Failed to establish a new connection: [Errno 8] nodename nor servname provided
, or not known',))

As shown above the ConnectionError is failing due to HTTPSConnectionPool(host='otx.alienvault.comapi', port=443) where the host variable is not a valid hostname.

After Patch:

>>> from OTXv2 import OTXv2
>>> otx = OTXv2(key)
>>> mirai_iocs = otx.get_pulse_indicators('58e39804b789b25b15083fc6', limit=100)
>>> len(mirai_iocs)
100

Issue retrieving single pulse

Making public record for issue (RS)

trying to write python script, got the following code:

initialize

from OTXv2 import OTXv2
from pandas.io.json import json_normalize
from datetime import datetime, timedelta
otx = OTXv2(my key)
otx.get("http://otx.alienvault.com/pulse/56e96f904637f2285c708f55")

keep getting the following error, no mattar how I format the url:

ValueError Traceback (most recent call last)
in ()
----> 1 otx.get("/pulse/56e71ff467db8c4088b184fe")
\OTX-Python-SDK\OTXv2.pyc in get(self, url)
66 response = None
67 try:
---> 68 response = request.open(url)
69 except URLError as e:
70 if e.code == 403:
C:\Anaconda\lib\urllib2.pyc in open(self, fullurl, data, timeout)
421
422 req.timeout = timeout
--> 423 protocol = req.get_type()
424
425 # pre-process request
C:\Anaconda\lib\urllib2.pyc in get_type(self)
283 self.type, self.__r_type = splittype(self.__original)
284 if self.type is None:
--> 285 raise ValueError, "unknown url type: %s" % self.__original
286 return self.type
287

ValueError: unknown url type: /pulse/56e96f904637f2285c708f55

I;m trying to download a specific pulse, how can I do it if not this way?
Thank you

getall API not working !

Here is my snippet:

from OTXv2 import OTXv2
from OTXv2 import IndicatorTypes

otx = OTXv2("xxxx")
pulses = otx.getall()
print(len(pulses))

the output:

Traceback (most recent call last):
  File "C:\Python39\lib\site-packages\requests\adapters.py", line 486, in send        
    resp = conn.urlopen(
  File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 878, in urlopen
    return self.urlopen(
  File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 878, in urlopen
    return self.urlopen(
  File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 878, in urlopen
    return self.urlopen(
  [Previous line repeated 2 more times]
  File "C:\Python39\lib\site-packages\urllib3\connectionpool.py", line 868, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "C:\Python39\lib\site-packages\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=50 (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python39\lib\site-packages\OTXv2.py", line 178, in get
    response = self.session().get(
  File "C:\Python39\lib\site-packages\requests\sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "C:\Python39\lib\site-packages\requests\sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python39\lib\site-packages\requests\sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python39\lib\site-packages\requests\adapters.py", line 510, in send
    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=50 (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Chipuku\Desktop\OTX automation\auto_otx.py", line 5, in <module>
    pulses = otx.getall()
  File "C:\Python39\lib\site-packages\OTXv2.py", line 404, in getall
    return self.walkapi(
  File "C:\Python39\lib\site-packages\OTXv2.py", line 385, in walkapi
    return list(self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body))
  File "C:\Python39\lib\site-packages\OTXv2.py", line 366, in walkapi_iter
    data = self.get(next_page_url)
  File "C:\Python39\lib\site-packages\OTXv2.py", line 187, in get
    raise RetryError()
OTXv2.RetryError: 'Exceeded maximum number of retries'

Why is that happend ?

Gameplay

Smooth gameplay playing, getting MVP

feature request: ability to submit threat indicators

As a feature request it would be beneficial to submit threat indictors via an API rather than only requesting.

No results retruned in get_indicator_details_full for IndicatorTypes.EMAIL

Code bellow returned empty for an AN_EMAIL_ADDRESS I have already found on

https://otx.alienvault.com/indicator/email/{{AN_EMAIL_ADDRESS}}

from OTXv2 import OTXv2
from OTXv2 import IndicatorTypes

API_KEY = "mykey_is_was_tested_an_was_ok"
AN_EMAIL_ADDRESS = "[email protected]"

otx = OTXv2(API_KEY)
found_entries = otx.get_indicator_details_full(IndicatorTypes.EMAIL, AN_EMAIL_ADDRESS)

print found_entries

Some more details:

• python 2.7.16
• packages on virtualenv:

$pip list
Package         Version
--------------- ---------
certifi         2020.12.5
chardet         4.0.0
idna            2.10
OTXv2           1.5.12
pip             20.3.4
pkg-resources   0.0.0
python-dateutil 2.8.1
pytz            2021.1
requests        2.25.1
setuptools      44.1.1
six             1.15.0
urllib3         1.26.4
wheel           0.36.2

Code branch master on 2021-04-15:

$ git status
On branch master
Your branch is up to date with 'origin/master'.
[...]

$ git config -l | grep remote.origin.url
remote.origin.url=https://github.com/AlienVault-OTX/OTX-Python-SDK

If necessary, please contact me to get the AN_EMAIL_ADDRESS address I have used do identify the problem.

Connecting from behind a proxy

in your OTXv2.py module, within the get function, you have the ProxyHandler set to http:

if self.proxy:
            proxy = ProxyHandler({'http': self.proxy})

If the user sets the proxy value, this will cause an error because the url being called out to is https:

server="https://otx.alienvault.com"

Updating the OTXv2.py module to 'https' rather than 'http' fixes the issue:

if self.proxy:
            proxy = ProxyHandler({'https': self.proxy})

Retrive All Pulses

Hello All,

As per Dashboard there are more that 4000 Pulses, but when I try to fetch all the pulses with below code
pulses = otx.getall(), I am able to fetch around 250 pulses only.

Does anyone knows how to fetch all the pulse in one go.

Thanks and Regards.

Add version method to API and call in OTXv2 init method

New feature request: please extend the API with a method to get the server's version. Also please consider invoking this method in the OTXv2 class init method and log the result at level debug.

In addition to yielding version details, this is a cheap way of testing if the API key is valid before trying to fetch data.

Thanks in advance for considering it.

Unicode decode error in OTXv2.py when calling "getall()"

With the new version of OTX-Python-SDK I get an error when calling getall(), which has not been there with the older version of the SDK.

It can be reproduced with my get-otx-iocs.py script and the right subscriptions.
https://github.com/Neo23x0/Loki/blob/master/threatintel/get-otx-iocs.py

As it happens in line 53 of OTXv2.py when decoding the whole response, it is diffucult to determin which pulse has the UNICODE characters that cannot be decoded.

prometheus:threatintel neo$ python get-otx-iocs_flo.py
Starting OTX feed download ...
Traceback (most recent call last):
  File "get-otx-iocs_flo.py", line 142, in <module>
    otx_receiver.get_iocs_last()
  File "get-otx-iocs_flo.py", line 50, in get_iocs_last
    self.events = self.otx.getall()
  File "/Library/Python/2.7/site-packages/OTXv2.py", line 62, in getall
    json_data = self.get(next)
  File "/Library/Python/2.7/site-packages/OTXv2.py", line 53, in get
    data = response.read().decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 128233: ordinal not in range(128)

EOF occurred in violation of protocol while Pulling pulses

Hello,

I am using pthon2.7, when I use
pulses = otx.getall()
It gives below error,
Traceback (most recent call last):
File "otx.py", line 7, in
pulses = otx.getall()
File "/usr/local/lib/python2.7/dist-packages/OTXv2.py", line 236, in getall
json_data = self.get(next_page_url)
File "/usr/local/lib/python2.7/dist-packages/OTXv2.py", line 81, in get
raise e
urllib2.URLError: <urlopen error [Errno 8] _ssl.c:510: EOF occurred in violation of protocol>

Anyone is aware of this issue, I tried the same with python3.4 as well still getting same error
Running ubuntu 14.04

Thanks,
Npormambi

Pulse Search Limitation

The search_pulses function only returns 5 results. Please add functionality to remove this limit when pulling large datasets.

events doesnt show pulses

events = otx.getevents_since(mtime)

it doesnt return pulses id.
Example:

 {
        "action": "unsubscribe",
        "created": "2015-12-08T19:40:05.337000",
        "id": "566732154637f27ede869c34",
        "object_id": "AlienVault",
        "object_type": "user"
    }

https://otx.alienvault.com/pulse/566732154637f27ede869c34/ - Pulse does not exist

Timeout Issue with get_indicator_details_full

When passing in anything other than URLs like hashes into otx.get_indicator_details_full for type IndicatorTypes.URL it hangs and there's no way to control timeout

Export API not supported anymore as per docs

The API: https://otx.alienvault.com/api/v1/indicators/export was earlier being used to fetch all the indicators. As per latest docs, the API isn't listed under https://otx.alienvault.com/assets/static/external_api.html#Home

Is the API removed? If so, what is the alternate API to use to fetch all indicators?
If this is not the correct repository to post in, please direct me to appropriate repository to create this issue in.

Pulling Private Pulses

Can the OTX python pull private pulses, or must they be public? The API key is tied to my account which created the private pulses.

Do I need to import from MISP to OTX one by one?

I am trying export all the event from MISP to .json and import into OTX Pulse. In the example folder only accept one .json file. Is there any method to bulk import all .json file into OTX?

Bug with IndicatorTypes.EMAIL

If I do res = otx.get_indicator_details_full(IndicatorTypes.EMAIL, args.EMAIL) I get the following error :

File "/home/user/env/analysis3/lib/python3.5/site-packages/OTXv2.py", line 490, in get_indicator_details_full
    for section in indicator_type.sections:
TypeError: 'NoneType' object is not iterable

Which makes sense because IndicatorTypes.EMAIL is the only one not having sections at all in IndicatorTypes.py while get_indicator_details_full use this to request information.

I have tried to fix this by adding a section but I end up having other errors :

~/env/analysis3/lib/python3.5/site-packages/OTXv2.py in get(self, url)
     81             else:
     82                 raise e
---> 83         data = response.read().decode('utf-8')
     84         json_data = json.loads(data)
     85         return json_data

AttributeError: 'NoneType' object has no attribute 'read'

I have this bug with python 3.5.2 and the pip package 1.2 (but I am pretty sure this bug is the master code)

get_my_pulses() function forms invalid API call URL, returns empty list.

In trying to use some simple python like this:

from OTXv2 import OTXv2

try:
   with open('config.json', 'r') as file:
      config = json.load(file)
except:
   quit('config.json not found, or unreadable.')

otx = OTXv2(config['otx_api_key'])

pulses = otx.get_my_pulses(max_items=200)
print(pulses)

The result would always be an empty list: [] No matter what arguments I passed to the get_my_pulses function.

I started digging into the source OTXv2 library. And hacked together the equivalent necessary functions using the requests library

from requests import Session
from requests.packages.urllib3.util import Retry
from requests.adapters import HTTPAdapter
import json

try:
   with open('config.json', 'r') as file:
      config = json.load(file)
except:
   quit('config.json not found, or unreadable.')


api = "https://otx.alienvault.com"
headers = {
            'X-OTX-API-KEY': config['otx_api_key'],
            'Content-Type': 'application/json'
        }

session = Session()
session.mount('https://', HTTPAdapter(
                max_retries=Retry(
                    total=5,
                    status_forcelist=[429, 500, 502, 503, 504],
                    backoff_factor=1,
                )
            ))

response = session.get(api+'/api/v1/pulses/my', headers=headers).json()
print(response['count'])

Which yielded a more expected response 214 which is accurate.

Upon digging more into the OTXv2 library functions, I hacked this together to give me just the necessary bits so that I could place some print statements and find what URLs were being formed.

import requests
try:
    from urllib.parse import urlencode
except ImportError:
    from urllib import urlencode
import json


API_V1_ROOT = "/api/v1"
MY_PULSES = "{}/pulses/my".format(API_V1_ROOT)
server="https://otx.alienvault.com"
headers = {
            'X-OTX-API-KEY': '<redacted>',
            'Content-Type': 'application/json'
        }

def post(url, body=None, headers=None, files=None, **kwargs):
    """
    Internal API for POST request on a OTX URL
    :param url: URL to retrieve
    :param body: HTTP Body to send in request
    :param headers: (optional) dict of headers to use, instead of default headers
    :param files: (optional) list of file tuples, if posting multipart form data
    :return: response as dict
    """
    response = requests.session().post(
        create_url(url, **kwargs),
        data=json.dumps(body) if body else None,
        files=files,
        headers=headers
    )
    print(json.dumps(response.json(), indent=2))
    return response.json()

def get(url, **kwargs):
    """
    Internal API for GET request on a OTX URL
    :param url: URL to retrieve
    :return: response in JSON object form
    """
    response = requests.session().get(create_url(url, **kwargs), headers=headers)
    print(json.dumps(response.json(), indent=2))
    return response.json()

def walkapi_iter(url, max_page=None, max_items=None, method='GET', body=None):
    next_page_url = url
    print('next_page_url', next_page_url)
    count = 0
    item_count = 0
    while next_page_url:
        count += 1
        if max_page and count > max_page:
            break
        if method == 'GET':
            data = get(next_page_url)
        elif method == 'POST':
            data = post(next_page_url, body=body)
        else:
            raise Exception("Unsupported method type: {}".format(method))
        for el in data['results']:
            item_count += 1
            if max_items and item_count > max_items:
                break
            yield el
        next_page_url = data["next"]

def create_url(url_path, **kwargs):
    """ Turn a path into a valid fully formatted URL. Supports query parameter formatting as well.
    :param url_path: Request path (i.e. "/search/pulses")
    :param kwargs: key value pairs to be added as query parameters (i.e. limit=10, page=5)
    :return: a formatted url (i.e. "/search/pulses")
    """
    print('----start create url----')
    uri = url_path.format(server)
    print(uri)
    uri = uri if uri.startswith("http") else server.rstrip('/') + uri
    print(uri)
    if kwargs:
        uri += "?" + urlencode(kwargs)
    print(uri)
    print('----stop create url---')
    return uri

def walkapi(url, iter=False, max_page=None, max_items=None, method='GET', body=None):
    if iter:
        print(walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body))
        return walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body)
    else:
        print(list(walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body)))
        return list(walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body))


def get_my_pulses(query=None, max_items=200):
    print('pulse?', walkapi(create_url(MY_PULSES, limit=50, q=query), max_items=max_items))
    #print(walkapi(create_url(MY_PULSES, limit=50)))
    return walkapi(create_url(MY_PULSES, limit=50, q=query), max_items=max_items)

This yielded URLs consistently like this, uring the create_url() function:

----start create url----
https://otx.alienvault.com/api/v1/pulses/my?limit=50&q=None
https://otx.alienvault.com/api/v1/pulses/my?limit=50&q=None
https://otx.alienvault.com/api/v1/pulses/my?limit=50&q=None
----stop create url---

q= is not a valid API query parameter according to the OTX API documentation: https://otx.alienvault.com/api

only limit, page, 'since`.

I'm haven't completely learned the OTXv2 code base, but I don't see anywhere were q= is supposed to be formatted into something the API accepts out of the above list of valid queries.

If you provide any sort of data by passing it in with get_my_pulses(query=<anything>) it always formats the URL as
https://otx.alienvault.com/api/v1/pulses/my?limit=50&q=<anything> which the API responds to with 0 results.

Am I missing something?

Note: I saw similar issues with get_user_pulses() but it appears to be virtually identical just with a different API endpoint URL.

Error 504 triying to get indicators

Hi:

Triying to use indicators and recieving error 504, I switch to basic testing with curl to debug the problem and still get 504

`curl "https://otx.alienvault.com/api/v1/indicators/export?modified_since=2022-02-25T12:35:00+00:00&limit=50&page=1" -H "X-OTX-API-KEY: XXX"

<title>504 Gateway Time-out</title>

504 Gateway Time-out

`

Not able to get results, don't know the proble, but it looks like something not working as expected.
Can anybody help here?

License

No license for this project has been defined, thus it is copyrighted by default. Therefore the code cannot be used by other project without violating copyright laws!

Limit returned fields

The endpoint GET /api/v1/pulses/{pulse_id} does not have the ability to limit the returned fields, or limit the number of indicators returned.

Example of not supporting limit to limit the indicators:

>>> pulse_id = '57eee565e65dd9042acfc830'
>>> pulse_url = otx.create_url(PULSE_DETAILS + str(pulse_id), limit=10)
>>> pulse_url
'https://otx.alienvault.com/api/v1/pulses/57eee565e65dd9042acfc830?limit=10&'
>>> pulse = otx.get(pulse_url)
>>> len(pulse['indicators'])
643

In the example, the pulse queried has 643 indicators, and even if I pass the parameter limit=10, I still get all 643.

I'd like to be able to limit the size of the response either by limiting the number of indicators returned, or by requesting specific fields to be only included in the response, for example:

otx.get('https://otx.alienvault.com/api/v1/pulses/57eee565e65dd9042acfc830?fields=name,author_name')

The reason I want to do this is because I want to be able to only get details of a pulse (for example, its name and author) by its pulse_id. I tried doing this using a search, but pulse IDs are not a supported query:

>>> otx.search_pulses('57ef3a9be65dd9042acfd1b0')
{u'count': 0, u'groups_count': 0, u'exact_match': u'', 'results': [], u'users_count': 0}

Is there any other way I can get the details of a pulse ID without having to pull all of its indicators?

Questionable 429 in status_forcelist

A question regarding OTXv2.session().
Please correct me if I'm wrong, but should 429 be included in status_forcelist? If the server says "too many requests", asking again in a second or half probably won't help.

self.request_session.mount('https://', HTTPAdapter(                                                                    
    max_retries=Retry(                                                                                                 
        total=5,                                                                                                       
        status_forcelist=[429, 500, 502, 503],                                                                         
        backoff_factor=1,                                                                                              
    )                                                                                                                  
))

Make a package

Currently it is necessary to copy the sdk file to software using this lib. It would be great if it would be a package, so we can just install it with pip and set it as requirement.

EDIT: I can help creating packages if needed, just did it with intelmq.

Python API is not returning whitelisted validation for IPv4

I used the below code to get indicator details for this IP address (8.8.8.8)
result = otx_client.get_indicator_details_by_section(IndicatorTypes.IPv4, '8.8.8.8', 'general')

But it doesn't return the whitelisted validation as in the example

If I use the other API, it returns the whitelisted validation successfully
validation = otx_client.validate_indicator(IndicatorTypes.IPv4, '8.8.8.8')

Example in README.md fails

This snippet of code from README.md fails in the google.com portion:

from OTXv2 import OTXv2
otx = OTXv2("API_KEY")
# Get all the indicators associated with a pulse
indicators = otx.get_pulse_indicators("pulse_id")
for indicator in indicators:
    print indicator["indicator"] + indicator["type"]
# Get everything OTX knows about google.com
otx.get_indicator_details_full(IndicatorTypes.DOMAIN, "google.com")

Needs the following additional line:

from OTXv2 import IndicatorTypes