susantabiswas/word-prediction-ngram

Next Word Prediction using n-gram Probabilistic Model with various Smoothing Techniques

Hi! I'm Susanta! 👋

• 🔭 Working on hyper-scale distributed systems.
• 🔭 Experienced in solving practical computer vision and NLP problems.
• 🌱 I’m currently learning more about distributed systems.

word-prediction-ngram's Issues

list index out of range

Hi,

There is no corpusfile.txt so i tried to test my txt file. First i get encoding error and i fixed it by adding encoding='utf-8' into loadCorpus.

Now i get another error. Its list index out of range. How can i fix it?

IndexError                                Traceback (most recent call last)
<ipython-input-17-c7bc734e5e35> in <module>()
      1 if __name__ == '__main__':
----> 2     main()

<ipython-input-16-abe7438bdb9e> in main()
     12     train_file = 'corpusfile.txt'
     13     #load the corpus for the dataset
---> 14     token_len = loadCorpus(train_file, bi_dict, tri_dict, quad_dict, vocab_dict)
     15 
     16     #estimate the lambdas for interpolation

<ipython-input-3-47bb9304cafa> in loadCorpus(file_path, bi_dict, tri_dict, quad_dict, vocab_dict)
     97 
     98             #store the last few words for the next sentence pairing
---> 99             w1 = token[n -3]
    100             w2 = token[n -2]
    101             w3 = token[n -1]

IndexError: list index out of range

resource references

Hey Susanta,
Do you have any theory references for the implementation, as I am not able to understand the notations used and also the purpose of some functions for eg: findBestFitIntercept, findBestFitSlope.
Thank you

Corpus File

Can you help me find the corpus file?