abdilahrf/ctfwriteupscrapper

Website to Scrapping all writeup from http://ctftime.org/ and you can organize which to read first!

CTFWriteupScrapper

Scrap all writeup from http://ctftime.org/ and organize which to read first.

Initialize SQLite DB:

python manage.py db init
python manage.py db migrate
python manage.py db upgrade

Start : python runserver.py

Open from browser 127.0.0.1:31337

Update your database with latest writeup from ctftime : 127.0.0.1:31337/feed/

Requirements

• Flask
• BeautifulSoup
• urllib2
• mechanize

Todo

Empty

Screenshot

License

MIT @abdilahrf

ctfwriteupscrapper's Issues

No module named 'flask_migrate'

$:~/Documents/ctf-abdilahrf/CTFWriteupScrapper# python manage.py db init
Traceback (most recent call last):
File "manage.py", line 2, in
from flask_migrate import Migrate, MigrateCommand
ModuleNotFoundError: No module named 'flask_migrate'

============================================================================
$:~/Documents/ctf-abdilahrf/CTFWriteupScrapper# pip install flask-script
Collecting flask-script
Using cached Flask-Script-2.0.6.tar.gz
Requirement already satisfied: Flask in /root/anaconda3/lib/python3.6/site-packages (from flask-script)
Requirement already satisfied: Werkzeug>=0.7 in /root/anaconda3/lib/python3.6/site-packages (from Flask->flask-script)
Requirement already satisfied: Jinja2>=2.4 in /root/anaconda3/lib/python3.6/site-packages (from Flask->flask-script)
Requirement already satisfied: itsdangerous>=0.21 in /root/anaconda3/lib/python3.6/site-packages (from Flask->flask-script)
Requirement already satisfied: click>=2.0 in /root/anaconda3/lib/python3.6/site-packages (from Flask->flask-script)
Requirement already satisfied: MarkupSafe>=0.23 in /root/anaconda3/lib/python3.6/site-packages (from Jinja2>=2.4->Flask->flask-script)
Building wheels for collected packages: flask-script
Running setup.py bdist_wheel for flask-script ... done
Stored in directory: /root/.cache/pip/wheels/35/38/2a/5a4aef4aa51913c135f5d2ff6f8552ad0db06667cd3e48ef2d
Successfully built flask-script
Installing collected packages: flask-script
Successfully installed flask-script-2.0.6

Unicode Errror during feeding

When I go to:
127.0.0.1:31337/feed/

File "/home/luigi/App/CTFWriteupScrapper/project/models.py", line 22, in init
self.original_writeup = str(data['originalWriteup'])
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 72: ordinal not in range(128)

I tried to fix it changing
self.original_writeup = str(data['originalWriteup'])
to
self.original_writeup = data['originalWriteup'].encode('utf-8')
but now I have this error

ProgrammingError: (sqlite3.ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. [SQL: u'INSERT INTO writeups (idwriteup, event, task, tags, author, ctftime_link, original_writeup) VALUES (?, ?, ?, ?, ?, ?, ?)'] [parameters: (7121, '/event/478|SHA2017 CTF', '/task/4426|Bon App?tit', 'web', u'/team/19216|HackXore', '/writeup/7121', 'https://github.com/chamli/Write_Up_Ctf/blob/master/SHA2017_CTF/Bon%20App\xc3\xa9tit.md')]

And I don't know how (and where) to change text_factory for sqlite

Filename

Just a minor issue. Filename typo. Yeah, I am a grammar nazi. Lol.
"requirments.txt" -> "requirements.txt"
"screenshoot.png" -> "screenshot.png"