vs666/data-scraping-and-analysis

Recruitment tasks for precog-2020

Precog-Submissions

Recruitment tasks for precog-2020

Submitted By

Varul Srivastava
B. Tech + MS (Computer Science,2024)
IIIT - Hyderabad

E-Mail : [email protected]
Website : http://vs666.github.io

Directory Structure of Submission

1. All README files contains reports
2. Directory structure is self explanatory, and in case of confusion, README files are there to clarify, (like in Q3)

Libraries used

1. tweepy
2. pandas
3. pymongo
4. json
5. csv
6. wordcloud (on command line)
7. matplotlib.pyplot
8. tabula-py
9. camelot
10. ExtractTable
11. pdfplumber
12. lxml

Note that ExtractTable does not come free, I have provided 2 API keys with 10 free attempts each. I incorporated this library despite limitations because of its superior quality of table extraction, using OpenCV and huge cloud dataset. So, in short, there are 20 attempts to choose that option, provided and for more, testers can register with their email address on the link provided.

MongoDB database couldnot be used properly due to multiple issues :

1. Unmet and unresolved dependencies on my machine
2. Lack of experience with mongoDB to resolve these errors in time, given I spent more effort on analysis of Data.