nullifiers / hackerrank-solution-crawler Goto Github PK
View Code? Open in Web Editor NEW๐ Crawls solutions of hackerrank and stores as local files.
License: MIT License
๐ Crawls solutions of hackerrank and stores as local files.
License: MIT License
Can we switch from travis to github actions instead ?
Pros-
Cons-
More on this https://knapsackpro.com/ci_comparisons/github-actions/vs/travis-ci
Apart from just running tests and build
other things can also be implemented
Increase code coverage by creating more unit tests and integration tests.
Currently, the script generates README files at the sub-domain level (Eg: Hackerrank/Algorithms/BitManipluation/). It'll be good if the script can similarly generate README files at root level (Hackerrank/) and domain level (Eg: Hackerrank/Algorithms/) as well.
A config file can be made to store different stuff like:
Currently sorting is happening from the wrong line number. We are splitting readme headers using \n
as the delimiter. So if there are 4 newlines, then headers' length will be 5 which should not be the case. That's why it is not considering the first row while sorting all the rows.
We can deploy on pypi automatically on tag release on master.
https://docs.travis-ci.com/user/deployment/pypi/
Right now, if two or more solutions to the same problem are found, then we override the existing solution with the new one crawled. But the crawling order is from newest to oldest. So, in this case, it will override the new solution with the old solution, which is not what we want.
Steps to reproduce the behavior:
Solution:
We also need to handle the offset and limit feature in this case.
Description
When downloading my submissions it returns the JSONDecodeError (see the code below).
The error is traced to the method login(username, password)
in line 45 of crawler.py.
As far I can figure out, the value of data
for data = resp.json()
is empty, resp.json() does not return anything.
Steps taken to try to fix
I could not figure out why resp.json()
returns nothing, so I tried writing my own crawler based on yours to simplify and understand what's going on.
My crawler worked fine enough in providing the authentication details and in getting the models or all the submissions, but when it came to calling resp.json() for each challenge in the get_submission(self, submissions)
method, it failed and retured the same JSONDecodeError.
If you wish, I can email my modified crawler to you.
Desktop:
Error message
user@pc:~$ hsc
Hackerrank Username: [email protected]
Hackerrank Password:
Traceback (most recent call last):
File "/home/parth/.local/bin/hsc", line 8, in <module>
sys.exit(main())
File "/home/parth/.local/lib/python3.8/site-packages/hsc/crawler.py", line 224, in main
if not crawler.authenticate():
File "/home/parth/.local/lib/python3.8/site-packages/hsc/crawler.py", line 65, in authenticate
return self.login(username, password)
File "/home/parth/.local/lib/python3.8/site-packages/hsc/crawler.py", line 45, in login
data = resp.json()
File "/usr/lib/python3/dist-packages/requests/models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 518, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
When i run hsc command after installation , it will asked for credentials to enter. I enter the correct details but it showing error.
Error:-
Traceback (most recent call last):
File "C:\Python27\Scripts\hsc-script.py", line 11, in
load_entry_point('hsc==1.2.1', 'console_scripts', 'hsc')()
File "c:\python27\lib\site-packages\hsc\crawler.py", line 221, in main
if not crawler.authenticate():
File "c:\python27\lib\site-packages\hsc\crawler.py", line 62, in authenticate
return self.login(username, password)
File "c:\python27\lib\site-packages\hsc\crawler.py", line 46, in login
self.get_number_of_submissions()
File "c:\python27\lib\site-packages\hsc\crawler.py", line 68, in get_number_of_submissions
self.total_submissions = resp.json()['total']
File "c:\python27\lib\site-packages\requests\models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "c:\python27\lib\json_init_.py", line 339, in loads
return _default_decoder.decode(s)
File "c:\python27\lib\json\decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "c:\python27\lib\json\decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
I also use the direct login command hsc -l 34 -p testpassword -u testuser
But this also not working!
Help me to resolve it!
There should be one test suite for the script, which executes the script and then checks if all the tasks which script needs to do are completed.
Run this test as part of Continous Integration.
Currently, the user has to wait for the completion of the crawler and then a message is displayed showing that "All solutions have been crawled".
It would be better if we can provide a progress bar showing the current progress of the number of solutions crawled.
https://stackoverflow.com/questions/3173320/text-progress-bar-in-the-console
Currently, there is no feature to directly fetch all the solutions user have done.
Firstly, we are fetching the total number of submissions user has done (to check whether login has been successful or not), so either we can show the number of submissions first, and then user can put the same number in the limit input to fetch all solutions.
On second thoughts, we can provide limit as a parameter to the script which will behave like this:
@rajat19 @rishabhsingh971 Please provide your opinions on this.
The script should run just by a command. For that some changes need to be done in the PyPi package.
newlines in written in quotes are printed as newline character (i.e they are not escaped)
Example:
line 31 should have been fptr.write(str(result) + '\n')
Cause:
https://github.com/Nullifiers/Hackerrank-Solution-Crawler/blob/master/hsc/crawler.py#L103
Why is this done @rajat19 @rajatgoyal715 ?
Currently login success is checked by comparing request URL with current URL which is not working.
https://github.com/Nullifiers/Hackerrank-Solution-Crawler/blob/master/hsc/crawler.py#L39
More file extensions should be added in the crawler.
I think we should separate input logic from crawler (in a different PR) or else it will become messy with time.
Originally posted by @rishabhsingh971 in #32
should we store some cookie instead ?
storing cookie is a good idea
we can also can cache the entire session. For that we can either manually pickle session or use https://github.com/rishabhsingh971/persession
Originally posted by @rishabhsingh971 in #23 (comment)
Currently All Problems listed in Readme are in order of their being crawled
Change code so that they are in ascending order of the name of Problem
Bug
Readme's generated do not contain valid solution links (path is incorrect)
Wrong path - /abc.cpp -> points to hsc/abc.cpp
Expected behavior
Expected path - ./abc.cpp -> points to hsc/Hackerrank/Algorithms/Implementation/abc.cpp
User should be able to pass/set flags through CLI.
Currently readme files created do not have language assigned with them.
This can cause confusions if a user has submitted successful solutions for a question using different langauge
What to do
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.