sjasthi / crawler Goto Github PK
View Code? Open in Web Editor NEWWeb Crawler for Telugu language words
License: MIT License
Web Crawler for Telugu language words
License: MIT License
The hosting is now done on bluehost.
The password reset mail needs to be properly configured.
It is working on the local host.
But not working on the remote host.
It might be due to the redirection.
Workaround is .. to explicitly type crawler.telugupuzzles.com/admin.php to see the admin options.
admin.php
crawl.php
parse.php
-- are missing on the tool bar as the LOGIN is not recognized on bluehost
While crawling URLs at multi-level, the following message is shown.
CRAWLING. Please wait, prompt will appear when finished.CRAWLING. Please wait, prompt will appear when finished.
However, it is not clear whether the system is stuck or whether it is still parsing the URLs.
Better interface/interaction will show the continuous progress.
One way to indicate that the operation is in progress is: indicating which URL is currently being processed.
Crawling started for http://url1......
200 unique words are inserted
157 existing words are ignored
Crawling ended for http://url1...
Crawling started for http://url2......
145unique words are inserted
34bexisting words are ignored
Crawling ended for http://url2...
All URLs are crawled.
6790 new words are inserted.
567 words are ignored.
Number of URLs processed in this crawl: 78
END.
Take the character "K". If you add vowels to this consonant, you get - ka, ke, ki, ko, ku
Searching for the base character "K" returns all the words k, ka, ke, ki, ko, ku. That is perfectly alright for English.
However, it doesn't work for Telugu.
In Telugu, all these are separate logical characters.
How to reproduce?
[1] Parse these words
కిలాడీలు
కోరికలు
కదలడు
కన్నతల్లి
[2] Set the MIN and MAX length to 4 (since we want only 4 letter words)
[3] Search for the PREFIX "క"
You should see only the words 3 and 4
[3] Search for the PREFIX "కి"
You should see only the 1st word
[3] Search for the PREFIX "కో"
You should see only the 2nd word.
However, in all the cases, you will now see all 4 words.
That is the bug.
https://crawler.telugupuzzles.com/telugu_characters_frequency_table.php?t=null
is showing the JUNK characters
UTF-8 characters are not getting exported correctly.
Tried the export from phpMyAdmin.
The DB / Table settings need to be changes so that the characters are getting exported correctly.
Parse method is not calculating the length of the words correctly.
However, if you take "Force Sync" operation, then the length is set correctly for each word.
https://crawler.telugupuzzles.com/stats.php
This is supposed to show the summary statistics on the crawler database
Searched the database for a Telugu character లి at the position 3.
(Give me all words between lengths 2 and 10 where the character లి is appearing at 3rd position).
It worked. Search results are shown. That is good.
However, I am also seeing a warning.
Notice: Uninitialized string offset: 2 in C:\xampp7371\htdocs\crawler\searchDB.php on line 170
Users searches for a string, or sub strings, or characters.
Results are displayed.
However, the search strings are all disappearing.
If the user wants to refine the search, he/she needs to start over.
So, the search string and the search parameters - these need to be preserved.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.