unkl4b / gitminer Goto Github PK
View Code? Open in Web Editor NEWTool for advanced mining for content on Github
License: GNU General Public License v3.0
Tool for advanced mining for content on Github
License: GNU General Public License v3.0
I'm getting the following error when using your docker setup:
python3: can't open file 'git_miner.py': [Errno 2] No such file or directory
Reproduction steps:
docker build -t gitminer .
docker run -it gitminer -h
{
"resource": "/e:/Gitminer/GitMiner/gitminer-v2.0.py",
"owner": "python",
"code": "anomalous-backslash-in-string",
"severity": 4,
"message": "Anomalous backslash in string: '\s'. String constant might be missing an r prefix.",
"source": "pylint",
"startLineNumber": 32,
"startColumn": 1,
"endLineNumber": 32,
"endColumn": 1
}
Its not showing nay result
Hi,
Your script is very helpful in searching for my company sensitive leakage on github.com, recently I face the problem that when I run the script as a thread or put it in a crontab for example:
python git_miner.py --query 'filename:configuration extension:php "public password" in:file' -m joomla -c pAAAhPOma9jEsXyLWZ-16RTTsGI8wDawbNs4 &
It's randomly terminated without an error message, I don't know if it is terminated by github.com or not? Do you have any idea?
Thanks.
requests.exceptions.SSLError: HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: /search?o=desc&q=filename:shadow%20path:etc&s=indexed&type=Code&utf8=%E2%9C%93 (Caused by SSLError(SSLError(1, u'[SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590)'),))
Traceback (most recent call last):
File "gitminer-v2.0.py", line 67, in
GitMiner().start()
File "gitminer-v2.0.py", line 39, in init
self.cookie = headers.parseCookie(self.args.cookie)
File "/root/GitMiner/config/headers.py", line 29, in parseCookie
values = line.strip().split("=")[1]
IndexError: list index out of range
请问这是出了什么问题
My operating system is centos6. Is it supported?
Great tool!
However I had to apply the following patch to make it work:
59c59
< self.cookie = {'user_session':self.args.cookie}
---
> self.cookie = dict(cookies_are=self.args.cookie)
Best regards,
Gwen
What do I put in it? I just don't know, Send me a picture or just describe it specificly where I can get it.
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:334: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:132: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecurePlatformWarning
Any help would be very appreciated,
Thanks.
Hey, I'm writing my Bachelor Thesis about "finding similarities in code from GitHub" . For this Bachelor Thesis I need a lot of code. Now I saw that your tool could help me to find a lot of code but all searches I try to type into your GitMiner Program didn't find any results. These are the searches I've typed in:
python git_miner.py -q 'extension:h #pragma omp in:file' -o result.txt
python git_miner.py -q 'extension:hpp #pragma omp in:file' -o result.txt
python git_miner.py -q 'extension:c #pragma omp in:file' -o result.txt
python git_miner.py -q 'extension:cpp #pragma omp in:file ' -o result.txt
python git_miner.py -q 'filename:* extension:h #pragma omp in:file ' -o result.txt
python git_miner.py -q 'filename:* extension:hpp #pragma omp in:file ' -o result.txt
python git_miner.py -q 'filename:* extension:c #pragma omp in:file ' -o result.txt
python git_miner.py -q 'filename:* extension:cpp #pragma omp in:file ' -o result.txt
python git_miner.py -q 'filename:test extension:h #pragma omp in:file ' -o result.txt
In his Program he's using the coockie from his own Github account, where can I find it?
Please help me than otherwise I have to open every github page and copy the code and download it.
Kind regards
Matthias Maltes
Hi Team,
I have tried to install these tools to do some manual checks on leakages of API key or any data for some organization for personal purposes, But while executing requirements.txt, we are prompted with version not supported, Then I have installed those dependencies via Python3 & pip3.
media/Tools/GitMiner# ls
config core Dockerfile gitminer-v2.0.py init.py LICENSE README.md requirements.txt
root@xx:/media/Tools/GitMiner# pip3 install requirements.txt
ERROR: Could not find a version that satisfies the requirement requirements.txt (from versions: none)
ERROR: No matching distribution found for requirements.txt
Please let me know, how to overcome these issues.
Hi,
I was wondering if you know a way how to sort all the data from the output.txt results of "-q filename:mup.json' -o output.txt -m mup" so it could read it in the following format:
HOST USER PASSWORD
right now it looks like (output.txt):
{"host": {"1": " "some.host.name",\r", "2": " false,\r"}, "password": {"1": " "somepassword@.!",\r"}}},{"@DTL625":
Thanks
with the command : $ python git_miner.py -q 'filename:wp-config extension:php FTP_HOST in:file' -m wordpress -o result.txt
██████╗ ██╗████████╗███╗ ███╗██╗███╗ ██╗███████╗██████╗
██╔════╝ ██║╚══██╔══╝████╗ ████║██║████╗ ██║██╔════╝██╔══██╗
██║ ███╗██║ ██║ ██╔████╔██║██║██╔██╗ ██║█████╗ ██████╔╝
██║ ██║██║ ██║ ██║╚██╔╝██║██║██║╚██╗██║██╔══╝ ██╔══██╗
╚██████╔╝██║ ██║ ██║ ╚═╝ ██║██║██║ ╚████║███████╗██║ ██║
╚═════╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝╚═╝╚═╝ ╚═══╝╚══════╝╚═╝ ╚═╝ v1.1
Automatic search for GitHub.
+ Autor: Danilo Vaz a.k.a. UNK
+ Blog: http://unk-br.blogspot.com
+ Github: http://github.com/danilovazb
+ Gr33tz: l33t0s, RTFM
+[WARNING]------------------------------------------+
| DEVELOPERS ASSUME NO LIABILITY AND ARE NOT |
| RESPONSIBLE FOR ANY MISUSE OR DAMAGE CAUSED BY |
| THIS PROGRAM |
+---------------------------------------------------+
Traceback (most recent call last):
File "git_miner.py", line 223, in <module>
GitMiner().start()
File "git_miner.py", line 216, in start
HTML = self.accessWeb(url_acesso)
File "git_miner.py", line 198, in accessWeb
if " find any code matching" in acc.text:
AttributeError: 'Response' object has no attribute 'text'
Same with other commands
[root@202-39-176-98 GitMiner]# python gitminer-v2.0.py --query 'filename:configuration extension:php "public password" in:file' -m joomla -c pAAAhPOma9jEsXyLWZ-16RTTsGI8wDawbNs4
Traceback (most recent call last):
File "gitminer-v2.0.py", line 67, in
GitMiner().start()
File "gitminer-v2.0.py", line 39, in init
self.cookie = headers.parseCookie(self.args.cookie)
File "/tmp/GitMiner/config/headers.py", line 29, in parseCookie
values = line.strip().split("=")[1]
IndexError: list index out of range
Hello,
Thanks for releasing new version, the older one is not working recently.
However, I've followed the installation guideline with below steps:
Traceback (most recent call last): File "gitminer-v2.0.py", line 11, in <module> from core.Parser import Parser File "/home/scumblr/GitMiner/core/Parser.py", line 6, in <module> from core.sendRequest import requestPage File "/home/scumblr/GitMiner/core/sendRequest.py", line 3, in <module> from core import Parser ImportError: cannot import name 'Parser'
Please help me, am I missing something?
Thanks.
root@localhost:~/GitMiner# python git_miner.py --query 'filename:configuration extension:php "public password" in:file' -m joomla
██████╗ ██╗████████╗███╗ ███╗██╗███╗ ██╗███████╗██████╗
██╔════╝ ██║╚══██╔══╝████╗ ████║██║████╗ ██║██╔════╝██╔══██╗
██║ ███╗██║ ██║ ██╔████╔██║██║██╔██╗ ██║█████╗ ██████╔╝
██║ ██║██║ ██║ ██║╚██╔╝██║██║██║╚██╗██║██╔══╝ ██╔══██╗
╚██████╔╝██║ ██║ ██║ ╚═╝ ██║██║██║ ╚████║███████╗██║ ██║
╚═════╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝╚═╝╚═╝ ╚═══╝╚══════╝╚═╝ ╚═╝ v1.1
Automatic search for GitHub.
Autor: Danilo Vaz a.k.a. UNK
Github: http://github.com/danilovazb
Gr33tz: l33t0s, RTFM
+[WARNING]------------------------------------------+
| DEVELOPERS ASSUME NO LIABILITY AND ARE NOT |
| RESPONSIBLE FOR ANY MISUSE OR DAMAGE CAUSED BY |
| THIS PROGRAM |
+---------------------------------------------------+
+[PAGE: 1/100]-----------------------------------------+
| [USER]: @monghimng
| [LINK]: http://github.com/monghimng/cmc/raw/d9a332d2cc22fc0b17ec9e93f8e5b917be382c83/configuration.php
Traceback (most recent call last):
File "git_miner.py", line 226, in
GitMiner().start()
File "git_miner.py", line 223, in start
self.parseSearch(HTML.content)
File "git_miner.py", line 179, in parseSearch
print("| [" + frescurinha.OKBLUE + "LAST INDEXED" + frescurinha.ENDC + "]: %s" % last_indexed[number_link])
IndexError: list index out of range
when running script
python git_miner.py -q 'filename:wp-config extension:php FTP_HOST in:file' -m wordpress -o result.txt
| [USER]: @cesarbgvieira
| [LINK]: http://github.com/cesarbgvieira/ProjectosAngola/raw/baaa4af8cdf7e075b39c494d88d93822b04ccd8c/citic/wp-config.php
| [LAST INDEXED]: Oct 10, 2015
| [CONTAIN]: "root" IN LINE: 22
| [PARAM FOUND]:
| -------------> FTP_USER
| -------------> César Vieira
Traceback (most recent call last):
File "git_miner.py", line 223, in <module>
GitMiner().start()
File "git_miner.py", line 220, in start
self.parseSearch(HTML.content)
File "git_miner.py", line 194, in parseSearch
self.nextPage(prox_page)
File "git_miner.py", line 73, in nextPage
self.parseSearch(HTML.content)
File "git_miner.py", line 194, in parseSearch
self.nextPage(prox_page)
File "git_miner.py", line 73, in nextPage
self.parseSearch(HTML.content)
File "git_miner.py", line 194, in parseSearch
self.nextPage(prox_page)
File "git_miner.py", line 73, in nextPage
self.parseSearch(HTML.content)
File "git_miner.py", line 194, in parseSearch
self.nextPage(prox_page)
File "git_miner.py", line 73, in nextPage
self.parseSearch(HTML.content)
File "git_miner.py", line 178, in parseSearch
self.parseCode(code_boladao)
File "git_miner.py", line 157, in parseCode
self.saveOutput("| -------------> %s\n" % result_code[i])
File "git_miner.py", line 66, in saveOutput
arquivo.write(text)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 18: ordinal not in range(128)
in page 5/15 (Mars the 3rd 2016)
Are you seriously putting the screeshot without processing the website address in the picture?
I ran GitMiner and it outputs nothing, just a small message with every example (in the Readme):
+ Autor: Danilo Vaz a.k.a. UNK
+ Blog: http://unk-br.blogspot.com
+ Github: http://github.com/danilovazb
+ Gr33tz: l33t0s, RTFM
+[WARNING]------------------------------------------+
| DEVELOPERS ASSUME NO LIABILITY AND ARE NOT |
| RESPONSIBLE FOR ANY MISUSE OR DAMAGE CAUSED BY |
| THIS PROGRAM |
+---------------------------------------------------+
+[PAGE: 1/1]-----------------------------------------+
I don't know why it worked last month and today it doesn't work.
Maybe Github made some changes on his search page.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.