evreny / python-web-crawler Goto Github PK
View Code? Open in Web Editor NEWThis project forked from kezakez/python-web-crawler
This project forked from kezakez/python-web-crawler
A programming excercise. A web crawler written in python. Usage python --version Python 2.7.2+ python crawl.py http://www.keza.net 2 JavaScript crawl.py [url] [depth] [search term] TODO the complex part. figure out how to do tests first. error handling. Thoughts It took 2 hours. I have never seen python before. python seems to make sense. finding the correct regex method was tricky. casting numbers to strings was annoying. Problem statement: Language: Python Official site: http://python.org/ Beginer's guide: http://wiki.python.org/moin/BeginnersGuide Tutorial: http://docs.python.org/tutorial/ App Category: Networking DB: None Simple part: Create a web crawler app in python which, given a url seed, can crawl through all links on the page and scan deep for a given level of depth. While crawling the app should be able to return the url page containing a specific search text. Input: 1 - Url seed e.g. www.hackernews.com 2 - Depth e.g. 5 (this means go into links on a page till 5 levels) 3 - search text e.g. "python" Output: the list of url that contains the specified text The Simple part is mandatory to be completed. If you finish the simple part and are eager to take up something challenging, then here's a little complex angle to the problem: Complex part: Write rules around the app for searching. Rule 1: The return Url should contain a specific substring Rule 2: Highlight in output if the url is amongst a long list of blacklisted urls (about 10000 blacklisted urls) Rule 3: Search for multiple search strings and rank Urls as per the number of different search strings found and occurances of each search string in the page Rule 4: Rank as per level of the Url w.r.t. seed url
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.