otx.getall() having error "Exceeded maximum number of retries"

[alienvault-otx]

otx.getall() having error "Exceeded maximum number of retries"

[Psr18delhi]

I was trying to extract my subscribed pulses but i am not able to download that. I am getting the above error. is there is any limit to download the subscribed or following pulses. I have 50K+ pulses in my dashboard is this is too large to download.

earlier i was able to download the 42K pulses. but this time i am getting the error. Please provide the solution.

[rustybrooks]

I didn't see this until now - are you still having problems with it? Can you post some of your code, or send it to me at [email protected] and I'll take a look? Please be careful not to post code with your API key in it!

[Psr18delhi]

@rustybrooks can you please help me to resolve this?

from OTXv2 import OTXv2, IndicatorTypes
from pandas import json_normalize
from datetime import datetime, timedelta, date
import json
import pandas

otx = OTXv2(API Key)
pulses = otx.getall()
events = otx.getsince(timestamp = "2023-02-23")

Till 10th Feb 2023. I was able to use the same functions and get the results. but then after I got the below error message.

Can you please help me to solve this problem?

---

MaxRetryError Traceback (most recent call last)
~\Anaconda3\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
448 retries=self.max_retries,
--> 449 timeout=timeout
450 )

~\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
845 body_pos=body_pos,
--> 846 **response_kw
847 )

~\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
845 body_pos=body_pos,
--> 846 **response_kw
847 )

~\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
845 body_pos=body_pos,
--> 846 **response_kw
847 )

~\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
845 body_pos=body_pos,
--> 846 **response_kw
847 )

~\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
845 body_pos=body_pos,
--> 846 **response_kw
847 )

~\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
818 try:
--> 819 retries = retries.increment(method, url, response=response, _pool=self)
820 except MaxRetryError:

~\Anaconda3\lib\site-packages\urllib3\util\retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
435 if new_retry.is_exhausted():
--> 436 raise MaxRetryError(_pool, url, error or ResponseError(cause))
437

MaxRetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20&modified_since=2023-02-23 (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

RetryError Traceback (most recent call last)
~\Anaconda3\lib\site-packages\OTXv2.py in get(self, url, **kwargs)
175 headers=self.headers,
--> 176 proxies=self.proxies,
177 )

~\Anaconda3\lib\site-packages\requests\sessions.py in get(self, url, **kwargs)
545 kwargs.setdefault('allow_redirects', True)
--> 546 return self.request('GET', url, **kwargs)
547

~\Anaconda3\lib\site-packages\requests\sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
532 send_kwargs.update(settings)
--> 533 resp = self.send(prep, **send_kwargs)
534

~\Anaconda3\lib\site-packages\requests\sessions.py in send(self, request, **kwargs)
645 # Send the request
--> 646 r = adapter.send(request, **kwargs)
647

~\Anaconda3\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
506 if isinstance(e.reason, ResponseError):
--> 507 raise RetryError(e, request=request)
508

RetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20&modified_since=2023-02-23 (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

RetryError Traceback (most recent call last)
in
----> 1 events = otx.getsince(timestamp = "2023-02-23")

~\Anaconda3\lib\site-packages\OTXv2.py in getsince(self, timestamp, limit, max_page, max_items)
418 """
419
--> 420 return self.getall(limit=limit, modified_since=timestamp, max_page=max_page, max_items=max_items, iter=False)
421
422 def getsince_iter(self, timestamp, limit=20, max_page=None, max_items=None):

~\Anaconda3\lib\site-packages\OTXv2.py in getall(self, modified_since, author_name, limit, max_page, max_items, iter)
393 return self.walkapi(
394 self.create_url(SUBSCRIBED, **args), iter=iter,
--> 395 max_page=max_page, max_items=max_items
396 )
397

~\Anaconda3\lib\site-packages\OTXv2.py in walkapi(self, url, iter, max_page, max_items, method, body)
372 return self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body)
373 else:
--> 374 return list(self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body))
375
376 def getall(self, modified_since=None, author_name=None, limit=20, max_page=None, max_items=None, iter=False):

~\Anaconda3\lib\site-packages\OTXv2.py in walkapi_iter(self, url, max_page, max_items, method, body)
353
354 if method == 'GET':
--> 355 data = self.get(next_page_url)
356 elif method == 'POST':
357 data = self.post(next_page_url, body=body)

~\Anaconda3\lib\site-packages\OTXv2.py in get(self, url, **kwargs)
178 return self.handle_response_errors(response).json()
179 except requests.exceptions.RetryError:
--> 180 raise RetryError()
181
182 def patch(self, url, body, **kwargs):

RetryError: 'Exceeded maximum number of retries'

[basvs]

We have the same problem;
library version: OTXv2-1.5.12

Everything worked until February 25th.

requests.exceptions.RetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=50&modified_since=2023-02-25T23%3A00%3A07.206000 (Caused by ResponseError('too many 504 error responses'))

We tried the library with both limit=20 and limit=50.

We also tried to request the URL with wget without the limit, but with the modified_since. That worked, but the library always sends the limit parameter..

[rholloway]

You are not alone @basvs . Not sure what's changed recently but the past couple weeks we have had very unstable results as well. Haven't made any changes on our end but getting lots of 504 recently.

[Psr18delhi]

@rustybrooks did you have any update?

[rustybrooks]

I haven't worked on OTX in well over a year. I don't actually know who's responsible for the OTX backend infrastructure these days. Sorry.

[Psr18delhi]

@chrisdoman any update?

[AnduinBrian]

i still have the same problem
my code is simple:

from OTXv2 import OTXv2
from OTXv2 import IndicatorTypes

otx = OTXv2("xxxx")
pulse = otx.getall(max_items=3, limit=5)
print(pulse) 

Even i set the limit but still have error: OTXv2.RetryError: 'Exceeded maximum number of retries'
The status code is 504. I think this api might dead lol