support for delay between request

[projectdiscovery]

support for delay between request

[0xAwali]

censys depend on cursor value to get next page e.g.

curl -s -u API-KEY:SECRET -H 'Content-Type: application/json' "https://search.censys.io/api/v2/hosts/search?q=company&per_page=100&cursor=base64-value" | jq -r .result.links.next

so token not issue here , I think it will be more efficiency to randomize the multiple tokens instead of one token every time because rate limit per seconds or minutes will not solve this issue if your result e.g. 100k because censys allow 250 queries if you are normal user so e.g. I was using this script to bypass the rate limit and allowed queries

#!/usr/bin/env bash
Query="services.tls.certificates.leaf_data.subject.organization:Facebook\bInc."
TokenCount=$(shuf -n 1 token.txt)
Count=$(curl -s -u $TokenCount -H 'Content-Type: application/json' "https://search.censys.io/api/v2/hosts/search?q=$Query&per_page=100" | jq -r .result.total)
Iters=$(expr "$Count" / 100 + 1)
Cursor=""
for ((I = 1 ; I <= "$Iters" ; I++))
do
	TokenSearch=$(shuf -n 1 token.txt)
	curl -s -u $TokenSearch -H 'Content-Type: application/json' "https://search.censys.io/api/v2/hosts/search?q=$Query&per_page=100&cursor=$Cursor" | jq -r .result.hits[].ip | tee -a result.txt
	sleep 1
	Cursor=$(curl -s -u $TokenSearch -H 'Content-Type: application/json' "https://search.censys.io/api/v2/hosts/search?q=$Query&per_page=100&cursor=$Cursor" | jq -r .result.links.next)
done

[ehsandeep]

@0xAwali thanks for creating this issue, about randomisation, it works based on execution, i.e new key will be picked on each execution of uncover and not per request as same key is required to rotate through multiple pages in case of results contains multiple results with higher limits.

[ehsandeep]

thanks for sharing more information @0xAwali around randomization improvements, and yes you are right about the rate limit, a default delay would be a better option to avoid the rate limit in general across all the engines.

[ehsandeep]

@0xAwali should be fixed on the dev version, now as default delay of 1 second is added.