Transliteration needs improvement

[quran]

Transliteration needs improvement

[mmahalwy]

This MAY help!

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/fuzzy-matching.html
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/phonetic-matching.html
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-phonetic-tokenfilter.html
https://github.com/elasticsearch/elasticsearch-analysis-phonetic

[ATouhou]

If you search for : "
http://alpha.quran.com/search?q=%D8%A7%D9%84%D8%B1%D9%82%D8%A7%D8%A8
"

Then the second result have " /em> " in the search results (in mid of the arabic text)

[goz]

@mmahalwy is it for translation index only? how is the ES indexing works currently, I see translation index and ayah index?

[mmahalwy]

@ATouhou yes getting that error not sure why. @gozali currently, ES indexes these models: https://github.com/quran/quran-api-rails/blob/dev/app/models/concerns/searchable.rb#L5 in which translation is further indexed by language.

This isn't too bad, but the challenge here is that transliteration is doing a match query rather than an actual transliteration refinement and match

[mmahalwy]

@gozali see what @ahmedre has built:

require 'levenshtein'

def compareText(ayahtext, query)
   str = ''

   best = 0
   cutoff = 80
   minPercent = 70

   # adjust percentage for short queries against long ayahs
   factor = ayahtext.length / query.length
   if (factor > 10)
      minPercent = minPercent - (factor / 2)
   end

   val = 0
   ayah = ayahtext
   while (true)
      if ayah.length == 1
         return 0
      end
      ayah = ayah[val..-1]

      if val == 0
         val = 1
      end

      if ((ayah[0] != query[0]) and (ayah.index(query[0]) == nil))
         return best
      end

      ayah = ayah[ayah.index(query[0])..-1]
      len = [ayah.length, query.length].min
      truncated = ayah[0..len]

print 'comparing ' + query + ' vs ' + truncated + "\n"
      distance = Levenshtein.distance query, truncated
print 'got ' + distance.to_s + "\n"
      percent = 100 * (1.0 - (distance.to_f / truncated.length))
print "percent: " + percent.to_s + "\n"
      if percent > minPercent
         if percent > best
            best = percent
         end

         if best > cutoff
            return best
         end
      end
   end

   return best
end

def prepareText(str)
   return str.gsub('oo', 'u')
             .gsub('-', '')
             .downcase
             .gsub('aa', 'a')
             .gsub(' ', '')
             .gsub('ia', 'i')
             .gsub('7', 'h')
end

ayah = 'bismillahirrahmanirraheem' # from db
print compareText(ayah, prepareText('ra7man'))

We need to run this either prior to ES to find matching ayahs OR somehow add this to ES for it to use this as a scoring mechanism for Transliteration. I don't know how to do that, but I think @sharabash would have a better idea.

[mmahalwy]

closing this