s0md3v/uro

declutters url lists for crawling/pentesting

Hi, I'm Somdev

I am a cyber security researcher and programmer.

Do you want to be one too? Check out my advice for learning hacking and programming.

You can support my work with a few bucks, here.

uro's Issues

Update to new version

Hi

I've installed the old version and how can I update to new version?
After installing the new version how can I know about the version? There isn't any help menu!

Please add more info about the update and help menu in readme file.

Thanks for sharing your code

Unable to install URO

I used pip command but not installed.

It shows :

Requirement already satisfied: uro in ./.local/lib/python3.11/site-packages (1.0.0)

It doesn't delete paths

When we check the paths, we see that 43935989 and 43935976 are used differently.

root@localhost:~# cat urls.txt
https://news.mail.ru/politics/43935976/?social=tw
https://news.mail.ru/politics/43935989/?social=tw

it should delete one of them but it doesn't.

root@localhost:~# cat urls.txt | uro
https://news.mail.ru/politics/43935976/?social=tw
https://news.mail.ru/politics/43935989/?social=tw

Can URO work on Windows

tried downloading URO on windows machine, is it possible to use uro on Wndows machine

Add support to decode different typees of encoded data

Multiple URL either have parameters with base64 encoding or any other encodings

Output formating

My suggestion is to get the output in json format, to tell if a particular url was decoded, and what encoding was present.

install issue

when i run this command (python3 setup.py install) it install but not working. OS=kali

Improvement Request

Hi Somdev,

1. I'd like to suggest you add the following extensions to be blacklisted. I have gathered all of these extensions manually and I think It would be nice to omit them:

'svg','img','gif','mp4','flv','ogv','webm','webp','mov','mp3','m4a','m4p','ppt','pptx','pdf','scss','tif','tiff','ttf','otf','woff','woff2','eot','htc','swf','rtf','image'

2. Also, I would like to ask for white-listing and allowing the js extension as there are lots of interesting features/endpoints to be found on them and I don't think if they are considered "useless".

Thanks!

Kind Regards,
HolyBugx

ERROR

i just can't get this to work
have cloned the repo and run the install command, bur when i try "cat file.txt | uro" it dosen't work.
do i have to do any additional commands?
any installation video??:)

uro error

cat urls.txt | uro > test

Traceback (most recent call last):
File "/usr/local/bin/uro", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/uro/uro.py", line 123, in main
for line in sys.stdin:
File "/usr/lib/python3.10/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

@s0md3v

URO for POST Request

Can i use URO for decluttering post request to use fetch unique requests using ffuf

Help

How Can I Get The Result In A txt File?

Command 'uro' not found

after installing
pip3 install uro when use this command
cat url.txt | uro it doesn't work as the photo

File filtering

Remove "js" file filtering

extend filtering

extend ext filter css|jpg|png|gif|svg|ico|woff|woff2|pdf|ttf|otf|jpeg|JPG|js|mp4|mp3|avi|eot

enhanced filtration

like i want to filter "/A/embed?url=" or "/B/embed?url=" which return similar data
like i want to filter "/A.php" or "/A.php/" which return similar data

broken pipe

I have been encountering this issue:

  File "/usr/local/bin/uro", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/uro/uro.py", line 151, in main
    print(host + path + dict_to_params(param))
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/bin/uro", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/uro/uro.py", line 161, in main
    print(host + path)
BrokenPipeError: [Errno 32] Broken pipe

Any idea why would it be?

--filters noexts Not working

More extension to declutter

Maybe it can be useful to add this extension to the one to declutter, at least, it's what I usually do:

.doc
.docx
.mp3
.mp4
.exe
.tif
.ttf
.woff
.woff2
.ico
.zip

Bad character range P-C at position 31

cat urls.txt | uro

Traceback (most recent call last):
  File "/usr/local/bin/uro", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/uro/uro.py", line 155, in main
    if re.search(pattern, path):
  File "/usr/lib/python3.8/re.py", line 201, in search
    return _compile(pattern, flags).search(string)
  File "/usr/lib/python3.8/re.py", line 304, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/lib/python3.8/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/lib/python3.8/sre_parse.py", line 948, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
  File "/usr/lib/python3.8/sre_parse.py", line 443, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/usr/lib/python3.8/sre_parse.py", line 598, in _parse
    raise source.error(msg, len(this) + 1 + len(that))
re.error: bad character range P-C at position 31

ImportError: cannot import name 'SIGPIPE' from 'signal'

D:\uro>uro
Traceback (most recent call last):
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\Scripts\uro-script.py", line 33, in
sys.exit(load_entry_point('uro==0.0.2', 'console_scripts', 'uro')())
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\Scripts\uro-script.py", line 25, in importlib_load_entry_point
return next(matches).load()
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\lib\importlib\metadata.py", line 77, in load
module = import_module(match.group('module'))
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\lib\importlib_init_.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 655, in _load_unlocked
File "", line 618, in _load_backward_compatible
File "", line 259, in load_module
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\lib\site-packages\uro-0.0.2-py3.8.egg\uro\uro.py", line 4, in
ImportError: cannot import name 'SIGPIPE' from 'signal' (C:\Users\umara\AppData\Local\Programs\Python\Python38\lib\signal.py)

Not checking similar web content

Hello,
This is not checking similar web content on many links. I will share the screenshot and URLs. Please fix it.

All of a sudden wont work

Traceback (most recent call last):
File "/usr/bin/uro", line 8, in
sys.exit(main())
^^^^^^
File "/home/greyhound/.local/lib/python3.11/site-packages/uro/uro.py", line 163, in main
process_url(parsed_url)
File "/home/greyhound/.local/lib/python3.11/site-packages/uro/uro.py", line 136, in process_url
new_params = [param for param in params.keys() if param not in params_seen]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/greyhound/.local/lib/python3.11/site-packages/uro/uro.py", line 136, in
new_params = [param for param in params.keys() if param not in params_seen]
^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

Ive tried tried setting aliases, changing paths etc wont work..

PLease solve this

└─# cat params.csv | uro | wc -l
Traceback (most recent call last):
File "/usr/local/bin/uro", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.9/dist-packages/uro/uro.py", line 155, in main
if re.search(pattern, path):
File "/usr/lib/python3.9/re.py", line 201, in search
return _compile(pattern, flags).search(string)
File "/usr/lib/python3.9/re.py", line 304, in _compile
p = sre_compile.compile(pattern, flags)
File "/usr/lib/python3.9/sre_compile.py", line 764, in compile
p = sre_parse.parse(p, flags)
File "/usr/lib/python3.9/sre_parse.py", line 962, in parse
raise source.error("unbalanced parenthesis")
re.error: unbalanced parenthesis at position 68
6547

Error

Traceback (most recent call last):
File "/usr/local/bin/uro", line 33, in
sys.exit(load_entry_point('uro==0.0.4', 'console_scripts', 'uro')())
File "/usr/local/bin/uro", line 25, in importlib_load_entry_point
return next(matches).load()
StopIteration

ValueError: Invalid IPv6 URL

cat all_urls.txt | uro

Traceback (most recent call last):
File "/usr/local/bin/uro", line 11, in
load_entry_point('uro==0.0.4', 'console_scripts', 'uro')()
File "/usr/local/lib/python3.10/dist-packages/uro-0.0.4-py3.10.egg/uro/uro.py", line 124, in main
File "/usr/lib/python3.10/urllib/parse.py", line 392, in urlparse
splitresult = urlsplit(url, scheme, allow_fragments)
File "/usr/lib/python3.10/urllib/parse.py", line 483, in urlsplit
raise ValueError("Invalid IPv6 URL")
ValueError: Invalid IPv6 URL

When using uro on subdomains it leaves :// in front?

When using uro on subdomains it leaves :// in front

example:

cat subs.txt | uro

subs.txt example:
site.com
sub.site.com
sub123.site.com

anything without http:// or https:// in front it leaves the :// in front.

Error install uro

suya has the error...
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behavior with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

I've done the steps above but haven't found a bright spot :(

can anyone help me???

I get a warning when installing uro with pip

Hello, in older versions of Kalin, I used to write 'pip3 install uro --user' and could directly use Uro. Now, I'm getting a warning as shown in the screenshot on SS and I can't use it. Can you help me with what I need to do?

false positive

hmmm ...

installed but not working

Uro error

λ cat newfile222.txt | uro
Traceback (most recent call last):
File "C:\Users\Yaseen\AppData\Local\Programs\Python\Python39\Scripts\uro-script.py", line 33, in
sys.exit(load_entry_point('uro==0.0.1', 'console_scripts', 'uro')())
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\site-packages\uro\uro.py", line 139, in main
if matches_patterns(path):
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\site-packages\uro\uro.py", line 107, in matches_patterns
if re.search(pattern, path):
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\re.py", line 201, in search
return _compile(pattern, flags).search(string)
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\re.py", line 304, in _compile
p = sre_compile.compile(pattern, flags)
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\sre_compile.py", line 764, in compile
p = sre_parse.parse(p, flags)
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\sre_parse.py", line 948, in parse
p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\sre_parse.py", line 443, in _parse_sub
itemsappend(_parse(source, state, verbose, nested + 1,
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\sre_parse.py", line 836, in _parse
raise source.error("missing ), unterminated subpattern",
re.error: missing ), unterminated subpattern at position 379
cat: write error: No space left on device

Can you help, it saying space issue, i have alot of space

error handling

So I added uro to my workflow and after a while I got this error:

Traceback (most recent call last):
  File "/usr/local/bin/uro", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/uro/uro.py", line 139, in main
    if matches_patterns(path):
  File "/usr/local/lib/python3.8/dist-packages/uro/uro.py", line 107, in matches_patterns
    if re.search(pattern, path):
  File "/usr/lib/python3.8/re.py", line 201, in search
    return _compile(pattern, flags).search(string)
  File "/usr/lib/python3.8/re.py", line 304, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/lib/python3.8/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/lib/python3.8/sre_parse.py", line 948, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
  File "/usr/lib/python3.8/sre_parse.py", line 443, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/usr/lib/python3.8/sre_parse.py", line 836, in _parse
    raise source.error("missing ), unterminated subpattern",
re.error: missing ), unterminated subpattern at position 369

It is happening to me with different inputs so seems to be something that happens often