Authors Search Engine using ElasticSearch and Python
- Download and Install the ElasticSearch
- Install the ICU_Tokenizer plugin on the ElasticSearch
- Install the python3 with pip3
- Install the python packages in the requirements.txt by pip install -r requirements.txt
- First start the ElasticSearch locally on port 9200.
- Then run index_creation.py file to create the index and insert data.
- Next run the main.py to start the search engine
- Then visit http://localhost:5000/ for see the user interface.
- Finally add your search query in the search box for searching.
- data - this folder contain authors datasheet and its json file, and python files of data cleaning
- templates - which includes HTML file of user interface
- documents - Folder contains project proposal & project report
- images - Folder contains diagrams used in README.md
- indexGeneration.py - Python file for index creating and data inserting
- main.py - main python file which handles API request of queries
- queryRequestJsonCreation.py - Elastic Search queries
- requirements.txt - txt file which has python dependancies
- searchCreation.py - Python file which process search request data
cleaned_authors.json file contains 103 Authors details with the following data.
- name - Author name in Sinhala
- name_english - Author name in English
- date_of_birth - date of birth of Author
- birth_place - Birth place of Author in Sinhala
- birth_place_english - Birth place of Author in English
- education - Educated places of Author
- languages - Lanugages is used to write
- categories - areas covered by writing
- list_of_books - books of Author
- description - Brief description of Author
- search query can be includes in any fields out of author_name, author_name_english, date_of_birth, birth_place, birth_place_english, languages, education, categories, list_of_books
eg: මහාචාර්ය එදිරිවීර සරච්චන්ද්ර, Arthur C. Clark, සිංහල, විද්යාප්රබන්ධ, Koggala
- there are synonym keyword lists to be identified special word tokens from search query. search will be customized according to priority of each field. whereas high weights will be given for identified fields by perticular keywords
eg: "මාර්ටින් වික්රමසිංහ රචකයා ගේ විස්තර" - word 'රචකයා' relates with AuthorName field. then authorName will be given high weight
- Search Engine supports bilingusl queries.
eg: එදිරිවීර සරච්චන්ද්ර රචකයා, author sarachchandra, author එදිරිවීර සරච්චන්ද්
- Search engine supports multiple field searches in search string
eg: "මළගිය ඇත්තෝ පොත රචනා කළ කතුවරයා" - 'පොත' relates with listOfBooks field, 'කතුවරයා' relates with AuthorName
Following figure shows how the search engine works through the flask server.
- 'ICU_Tokenizer’ which is a standard tokenizer and which has better support for sinhala language to tokenize text into the words.
- Elastic search ‘edge_ngram’ filter was used to generate n-grams.
-
Rule-based text mining is used to understand and extract data from the user entered query string.
-
A basic set of rules are applied to each search phrase to identify the keywords, classify them into relevant search types. According to the classification user query was classified to one of the following query type.
- Faceted query with Cross Fields
- Faceted query with phrase-prefix
Follwing diagram further shows the use of Rule Based Classification and Multisearch queries.