vulsio / go-exploitdb Goto Github PK
View Code? Open in Web Editor NEWTool for searching Exploits from Exploit Databases, etc.
License: MIT License
Tool for searching Exploits from Exploit Databases, etc.
License: MIT License
CVE 2018 feed xml keeps unable to be parsed. (Maybe #18 is the same)
$ go-exploitdb fetch exploitdb
...
WARN[08-02|02:24:01] Failed to Unmarshal XML. URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2018.xml err="XML syntax error on line 169238: expected /> in element"
I tried 4 times curl -O http://cve.mitre.org/data/downloads/allitems-cvrf-year-2018.xml
, but all of them were broken at around 10MB (response body ended with "Virus/Spyware Download Blocked" html message).
Then I tried changing protocol http to https curl -O https://cve.mitre.org/data/downloads/allitems-cvrf-year-2018.xml
, it was all OK (22MB).
Feed urls in the CVE site use https https://cve.mitre.org/data/downloads/index.html .
Currently go-exploitdb uses http when downloading CVE feeds, but it might be good to use https.
It takes over 10 min to fetch with -deep mode.
I think it can speed up fetching.
Hi, now that #112 has merged and the old GitHub repo was archived and emptied go-exploitdb fetch exploitdb
no longer works.
This should warrant a v0.4.3 release? @MaineK00n
Thanks
Hi got error when i wanted to make install this repo .
please help me
[root@localhost go-exploitdb]# make install
fatal: No names found, cannot describe anything.
go install -ldflags "-X 'main.version=' "
Hello!
I tried to get your code, but i have the error:
go get github.com/knqyf263/gost
package github.com/labstack/echo/engine/standard: cannot find package "github.com/labstack/echo/engine/standard" in any of:
/usr/local/go/src/github.com/labstack/echo/engine/standard (from $GOROOT)
/home/alexander/go/src/github.com/labstack/echo/engine/standard (from $GOPAT
github.com/labstack/echo/engine/standard - 404 Not found.
Could your check it please?
Thanks
please build standart release for downloading instead of building project on every server
There are exploit codes which are not registered in exploit-db, so it is better to collect them.
ex)
<Vulnerability xmlns="http://www.icasi.org/CVRF/schema/vuln/1.1" Ordinal="312">
<Title>CVE-1999-0313</Title>
<Notes><Note Type="Description" Ordinal="1">disk_bandwidth on SGI IRIX 6.4 S2MP for Origin/Onyx2 allows local
users to gain root access using relative pathnames.
</Note>
<Note Type="Other" Ordinal="2" Title="Published">1999-09-29</Note>
<Note Type="Other" Ordinal="3" Title="Modified">2005-11-02</Note>
</Notes>
<CVE>CVE-1999-0313</CVE>
<References>
<Reference>
<URL>http://www.securityfocus.com/bid/213/exploit</URL>
<Description>MISC:http://www.securityfocus.com/bid/213/exploit</Description>
</Reference>
<Reference>
<URL>ftp://patches.sgi.com/support/free/security/advisories/19980701-01-P</URL>
<Description>SGI:19980701-01-P</Description>
</Reference>
<Reference>
<URL>http://www.securityfocus.com/bid/214</URL>
<Description>BID:214</Description>
</Reference>
<Reference>
<URL>http://www.osvdb.org/936</URL>
<Description>OSVDB:936</Description>
</Reference>
<Reference>
<URL>https://exchange.xforce.ibmcloud.com/vulnerabilities/1441</URL>
<Description>XF:sgi-disk-bandwidth(1441)</Description>
</Reference>
</References>
</Vulnerability>
securityfocus has a exploit-code of CVE-1999-0313 http://www.securityfocus.com/bid/213/exploit
but exploit-db doesn't has it.
I launched go-exploitdb in docker environment using following docker-compose.yml
:
version: '3'
services:
# https://github.com/vulsio/go-exploitdb
go-exploitdb:
image: vuls/go-exploitdb
command: server --debug
ports:
- 1236:1236
volumes:
- ./:/vuls
- ./vuls-log:/var/log/vuls
and launch docker like: docker-compose --log-level DEBUG up
Result of docker ps:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d1353354fbc9 vuls/go-exploitdb "go-exploitdb server…" 10 minutes ago Up 1 second 0.0.0.0:1236->1236/tcp, :::1236->1236/tcp ludo_go-exploitdb_1
go-exploitdb fetch commands works fine.
When i try: curl http://192.168.1.41:1326/cves/CVE-2014-0160
I receive following error message:
curl: (7) Failed to connect to 192.168.1.41 port 1326: Connection refused
Go version (go version
): go version go1.13.8 linux/amd64
Go environment (go env
):
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go-1.13"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.13/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build142440163=/tmp/go-build -gno-record-gcc-switches"
Hash : ____
To check the commit hash of HEAD
$ go-exploitdb version
--> No version information
or
$ cd $GOPATH/src/github.com/vulsio/go-exploitdb
$ git rev-parse --short HEAD
--> Command failed
NOTE:
It works fine using:
docker exec go-exploitdb go-exploitdb search --type CVE --param CVE-2009-4091
but fails using:
curl http://192.168.1.41:1326/cves/CVE-2009-4091
Any idea ?
Regards,
Everyday I update go-exploitdb (git pull; rm -r vendor; make install
) with cron. After 2019-1-12, "go-exploitdb fetch" command begun to fail with "FOREIGN KEY constraint failed" error message."
Here is the error message from grep eror /var/log/go-exploitdb/go-exploitdb.log
.
t=2019-01-09T04:27:50+0900 lvl=eror msg="Failed to fetch Exploit" err="empty csv file given"
t=2019-01-10T04:27:09+0900 lvl=eror msg="Failed to fetch Exploit" err="empty csv file given"
t=2019-01-11T04:29:23+0900 lvl=eror msg="Failed to fetch Exploit" err="empty csv file given"
t=2019-01-12T04:30:04+0900 lvl=eror msg="Failed to insert." dbpath= err="Failed to delete old records. err: FOREIGN KEY constraint failed"
t=2019-01-13T04:31:24+0900 lvl=eror msg="Failed to insert." dbpath= err="Failed to delete old records. err: FOREIGN KEY constraint failed"
t=2019-01-14T04:29:07+0900 lvl=eror msg="Failed to insert." dbpath= err="Failed to delete old records. err: FOREIGN KEY constraint failed"
t=2019-01-15T04:29:37+0900 lvl=eror msg="Failed to insert." dbpath= err="Failed to delete old records. err: FOREIGN KEY constraint failed"
t=2019-01-15T19:04:13+0900 lvl=eror msg="Failed to insert." dbpath= err="Failed to delete old records. err: FOREIGN KEY constraint failed"
2019-01-09 : 2019-01-11 errors may be related with #13 issue.
2019-01-12 : 2019-01-15 errors are related with this issue.
cf. related twitter conversation (in Japanese language)
https://twitter.com/tako774/status/1085117456346017794
This might be related to #17 except I am not using a proxy at all. I tried two different internet connections with the same results. I can browse http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml via firefox and download via wget every time but when I use go-exploitdb it times out every time.
$ go-exploitdb fetch exploitdb --debug
INFO[06-17|18:05:02] Opening Database. db=sqlite3
INFO[06-17|18:05:02] Migrating DB. db=sqlite3
INFO[06-17|18:05:02] Fetching Offensive Security Exploit
INFO[06-17|18:05:02] Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml
EROR[06-17|18:21:07] Failed to fetch Exploit err="Failed to fetch cve data from Mitre. targetURL: http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml: HTTP error. errs: [Get http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml: read tcp 192.168.1.10:44626->192.52.194.135:80: read: connection timed out], url: http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml"
Failed to fetch cve data from Mitre. targetURL: http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml: HTTP error. errs: [Get http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml: read tcp 192.168.1.10:44626->192.52.194.135:80: read: connection timed out], url: http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml
Any thoughts on what I can do to fix this? Thanks.
Cannot fetch properly using docker:
docker run --rm princechrismc/go-exploitdb fetch
Log output:
t=2019-01-08T16:32:22+0000 lvl=info msg="Initialize Database"
t=2019-01-08T16:32:22+0000 lvl=info msg="Opening Database." db=sqlite3
t=2019-01-08T16:32:22+0000 lvl=info msg="Fetching Exploit"
t=2019-01-08T16:32:22+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml
t=2019-01-08T16:32:22+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2000.xml
t=2019-01-08T16:32:23+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2001.xml
t=2019-01-08T16:32:23+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2002.xml
t=2019-01-08T16:32:25+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2003.xml
t=2019-01-08T16:32:26+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2004.xml
t=2019-01-08T16:32:27+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2005.xml
t=2019-01-08T16:32:28+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2006.xml
t=2019-01-08T16:32:30+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2007.xml
t=2019-01-08T16:32:32+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2008.xml
t=2019-01-08T16:32:34+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2009.xml
t=2019-01-08T16:32:35+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2010.xml
t=2019-01-08T16:32:37+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2011.xml
t=2019-01-08T16:32:39+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2012.xml
t=2019-01-08T16:32:41+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2013.xml
t=2019-01-08T16:32:43+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2014.xml
t=2019-01-08T16:32:45+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2015.xml
t=2019-01-08T16:32:47+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2016.xml
t=2019-01-08T16:32:49+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2017.xml
t=2019-01-08T16:32:51+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2018.xml
t=2019-01-08T16:32:54+0000 lvl=info msg=Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2019.xml
t=2019-01-08T16:32:55+0000 lvl=info msg=Fetching URL=https://raw.githubusercontent.com/offensive-security/exploitdb/master/files_shellcodes.csv
t=2019-01-08T16:32:55+0000 lvl=info msg=Fetching URL=https://raw.githubusercontent.com/offensive-security/exploitdb-papers/master/files_papers.csv
t=2019-01-08T16:32:55+0000 lvl=eror msg="Failed to fetch Exploit" err="empty csv file given"
Hello,
is it possible to use a angeric search pattern e.g. like a regex similar the command searchsploit openssh 6.8
. In my case I have got the product name and a version and would like to get all existing CVEs for this combination
try to fetch latest githubrepos
Download exploits from githubrepos source
error in pocsrc clone or pull
All other sources works fine.
Regards
This issue is an enhancement request. Could the image run as a non-root user?
Currently, the image runs as the root user.
$ docker run --rm -it --entrypoint=/bin/sh vuls/go-exploitdb:0.1.4
/vuls # id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
/vuls #
If the user is a non-root user, we would be able to follow best practices of running as a non-root user (e.g. [1] [2]) and add proper restrictions in Pod Security Policies.
Integrate the Fetcher and Search option into my program by using the SQLite database. I would like to use go-exploitdb
as library in an own program with SQLite database for offline usage
I get an error batch-size option is not set properly
See the line the batch-size is set by an environment variable / argument flag and not by a property of the RDBDriver
struct.
Add to the Option struct a property batchSize: uint
and use it instead of the viper.GetInt("batch-size")
call and it would be nice to provide a default value.
go version
): 1.22The --http-proxy option in go-exploitdb does not seem to work properly. Do I need to edit code?
I applied the same thing as the --http-proxy option I used at the gost patch, but the proxy does not seem to work properly.
INFO[02-04|14:20:32] Initialize Database
INFO[02-04|14:20:32] Opening Database. db=sqlite3
INFO[02-04|14:20:32] Fetching Exploit
INFO[02-04|14:20:32] Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml
EROR[02-04|14:22:39] Failed to fetch Exploit err="Failed to fetch cve data from Mitre. targetURL: http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml: HTTP error. errs: [Get http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml: dial tcp 198.49.146.233:80: connect: connection timed out], url: http://cve.mitre.org/data/downloads/allitems-cvrf-year-1999.xml"
run go-exploitdb fetch inthewild
with postgres backend
The database to be successfully fetched and stored in postgres
Failed to insert. dbpath: service=vuls, err: Failed to insert. err: ERROR: value too long for type character varying(255) (SQLSTATE 22001)
see above
No specific configuration besides postgres-specific settings in ~/.pg_service.conf
go-exploitdb --dbtype=postgres --dbpath=service=vuls fetch inthewild
go version
):go version go1.19.4 freebsd/amd64
To check the commit hash of HEAD
$ go-exploitdb version
go-exploitdb v0.4.2 4d40a8e
Issue appears to be at
go-exploitdb/models/exploit.go
Line 29 in 4d40a8e
Verified from source sqlite db:
curl -LO https://github.com/gmatuz/inthewilddb/raw/master/inthewild.db
sqlite3 inthewild.db 'select count(*) from exploits where length(referenceURL) > 255'
13
I am getting the following error.
Failed to fetch InTheWild Exploit. err: Failed to fetch inTheWild DB. err: HTTP error. URL: https://github.com/gmatuz/inthewilddb/blob/master/inthewild.db?raw=true, err: []
I think there are some changes in inthewilddb repo and the link returns 404 now.
I keep getting this:
INFO[02-23|13:37:34] Fetching URL=http://cve.mitre.org/data/downloads/allitems-cvrf-year-2017.xml
EROR[02-23|13:37:40] Failed to fetch Exploit err="Failed to Unmarshal XML: XML syntax error on line 207064: element <meta> closed by </head>"
anybody else has that problem ?
"go-exploitdb server" do not use default port but a random one.
"go-exploitdb server -port" option do not apply specified port.
$ kubectl exec -it go-exploitdb-7fd57d6fc5-bw9bq go-exploitdb server
INFO[04-11|14:35:40] Opening Database. db=sqlite3
INFO[04-11|14:35:40] Migrating DB. db=sqlite3
INFO[04-11|14:35:40] Starting HTTP Server...
INFO[04-11|14:35:40] Listening... URL=:
____ __
/ __/___/ / ___
/ _// __/ _ \/ _ \
/___/\__/_//_/\___/ v3.3.10-dev
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
O\
⇨ http server started on [::]:37709
$ kubectl exec -it go-exploitdb-7fd57d6fc5-bw9bq go-exploitdb server
INFO[04-11|14:35:42] Opening Database. db=sqlite3
INFO[04-11|14:35:42] Migrating DB. db=sqlite3
INFO[04-11|14:35:42] Starting HTTP Server...
INFO[04-11|14:35:42] Listening... URL=:
____ __
/ __/___/ / ___
/ _// __/ _ \/ _ \
/___/\__/_//_/\___/ v3.3.10-dev
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
O\
⇨ http server started on [::]:45149
$ kubectl exec -it go-exploitdb-7fd57d6fc5-bw9bq -- go-exploitdb server --bind 0.0.0.0 --port 1326
INFO[04-11|14:35:30] Opening Database. db=sqlite3
INFO[04-11|14:35:30] Migrating DB. db=sqlite3
INFO[04-11|14:35:30] Starting HTTP Server...
INFO[04-11|14:35:30] Listening... URL=:
____ __
/ __/___/ / ___
/ _// __/ _ \/ _ \
/___/\__/_//_/\___/ v3.3.10-dev
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
O\
⇨ http server started on [::]:35895
anyway great job ! Nice appli
Best regards
When trying to pull githubrepos information, go-expliotdb is failing to sleep. It says it is in the output, but it instantly jumps to the next line.
$ go-exploitdb fetch --debug githubrepos
INFO[06-17|17:43:26] Opening Database. db=sqlite3
INFO[06-17|17:43:26] Migrating DB. db=sqlite3
INFO[06-17|17:43:26] Fetching GitHub Repos Exploit
INFO[06-17|17:43:26] Fetching GitHub Repository year=1999
INFO[06-17|17:43:27] Fetching GitHub Repository year=2000
INFO[06-17|17:43:27] Fetching GitHub Repository year=2001
INFO[06-17|17:43:28] Fetching GitHub Repository year=2002
INFO[06-17|17:43:29] Fetching GitHub Repository year=2003
INFO[06-17|17:43:30] Fetching GitHub Repository year=2004
INFO[06-17|17:43:31] Fetching GitHub Repository year=2005
INFO[06-17|17:43:32] Fetching GitHub Repository year=2006
INFO[06-17|17:43:33] Fetching GitHub Repository year=2007
INFO[06-17|17:43:33] Sleep for GitHub rate limit duration=-24m12.825496102s
INFO[06-17|17:43:33] Fetching GitHub Repository year=2008
INFO[06-17|17:43:34] Sleep for GitHub rate limit duration=-24m13.613048149s
INFO[06-17|17:43:34] Fetching GitHub Repository year=2009
EROR[06-17|17:43:35] Failed to fetch GitHubRepo Exploit err="HTTP error. errs: [], status code: 403, url: https://api.github.com/search/repositories?q=CVE%202009+in:name&&page=1&per_page=100"
HTTP error. errs: [], status code: 403, url: https://api.github.com/search/repositories?q=CVE%202009+in:name&&page=1&per_page=100
This is a Red Hat 7 (fully patched/updated) system. Just to verify I did a fresh pull with commit a055cc2 and still have the problem.
.\go-exploitdb.exe fetch githubrepos
INFO[09-07|17:27:46] Fetching GitHub Repos Exploit
Failed to fetch GitHubRepo Exploit. err: error in pocsrc clone or pull: Failed to pull repository: Failed to open repository: repository does not exist
.\go-exploitdb.exe fetch inthewild
INFO[09-07|18:03:42] Fetching inTheWild Poc Exploit
Failed to fetch InTheWild Exploit. err: Failed to fetch inTheWild DB. err: HTTP error. url: https://github.com/gmatuz/inthewilddb/blob/master/inthewild.db?raw=true, err: []
$ /home/vulsuser/go/bin/go-exploitdb fetch exploitdb --dbpath=/home/vulsuser/go-exploitdb.sqlite3
INFO[11-14|11:56:51] Fetching Offensive Security Exploit
INFO[11-14|11:56:51] Fetching URL=https://cve.mitre.org/data/downloads/allitems-cvrf.xml
INFO[11-14|11:58:02] Fetching URL=https://raw.githubusercontent.com/offensive-security/exploitdb/master/files_shellcodes.csv
Failed to fetch Exploit. err: HTTP error. url: https://raw.githubusercontent.com/offensive-security/exploitdb/master/files_shellcodes.csv, err: []
Go version (go version
):
$ go version
go version go1.18.3 linux/amd64
To check the commit hash of HEAD
$ go-exploitdb version
go-exploitdb v0.4.2 7bfd613
$ cd $GOPATH/src/github.com/vulsio/go-exploitdb
$ git rev-parse --short HEAD
7bfd613
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.