sjasthi/crawler

Web Crawler for Telugu language words

crawler's Issues

(Bug) Password reset is going to a different host

The hosting is now done on bluehost.
The password reset mail needs to be properly configured.

(Bug) logging in as admin on bluehost not working

It is working on the local host.
But not working on the remote host.
It might be due to the redirection.

Workaround is .. to explicitly type crawler.telugupuzzles.com/admin.php to see the admin options.

admin.php
crawl.php
parse.php
-- are missing on the tool bar as the LOGIN is not recognized on bluehost

(Enhancement) Meaningful progress indicator

While crawling URLs at multi-level, the following message is shown.

CRAWLING. Please wait, prompt will appear when finished.CRAWLING. Please wait, prompt will appear when finished.

However, it is not clear whether the system is stuck or whether it is still parsing the URLs.
Better interface/interaction will show the continuous progress.

One way to indicate that the operation is in progress is: indicating which URL is currently being processed.

Crawling started for http://url1......
200 unique words are inserted
157 existing words are ignored
Crawling ended for http://url1...

Crawling started for http://url2......
145unique words are inserted
34bexisting words are ignored
Crawling ended for http://url2...

All URLs are crawled.
6790 new words are inserted.
567 words are ignored.
Number of URLs processed in this crawl: 78
END.

searching for the base character క is not working

Take the character "K". If you add vowels to this consonant, you get - ka, ke, ki, ko, ku
Searching for the base character "K" returns all the words k, ka, ke, ki, ko, ku. That is perfectly alright for English.

However, it doesn't work for Telugu.
In Telugu, all these are separate logical characters.

How to reproduce?

[1] Parse these words
కిలాడీలు
కోరికలు
కదలడు
కన్నతల్లి

[2] Set the MIN and MAX length to 4 (since we want only 4 letter words)

[3] Search for the PREFIX "క"
You should see only the words 3 and 4

[3] Search for the PREFIX "కి"
You should see only the 1st word

[3] Search for the PREFIX "కో"
You should see only the 2nd word.

However, in all the cases, you will now see all 4 words.
That is the bug.

(Bug) character count is showing the junk characters (not Telugu characters)

https://crawler.telugupuzzles.com/telugu_characters_frequency_table.php?t=null

is showing the JUNK characters

Exporting "telugu" table is showing junk characters in CSV

UTF-8 characters are not getting exported correctly.

Tried the export from phpMyAdmin.

The DB / Table settings need to be changes so that the characters are getting exported correctly.

(Bug) Parse vs Force Sync

Parse method is not calculating the length of the words correctly.
However, if you take "Force Sync" operation, then the length is set correctly for each word.

(Regression) Summary - is not working

https://crawler.telugupuzzles.com/stats.php

This is supposed to show the summary statistics on the crawler database

• English words
• Eelugu words
• crawled URLs, etc.
  But it has regressed. Nothing is showing up.

"Contains Character at.." is throwing an error.

Searched the database for a Telugu character లి at the position 3.

(Give me all words between lengths 2 and 10 where the character లి is appearing at 3rd position).

It worked. Search results are shown. That is good.
However, I am also seeing a warning.

Notice: Uninitialized string offset: 2 in C:\xampp7371\htdocs\crawler\searchDB.php on line 170

Search string is disappearing after the results are displayed

Users searches for a string, or sub strings, or characters.

Results are displayed.

However, the search strings are all disappearing.

If the user wants to refine the search, he/she needs to start over.

So, the search string and the search parameters - these need to be preserved.