There is no corpusfile.txt so i tried to test my txt file. First i get encoding error and i fixed it by adding encoding='utf-8' into loadCorpus.
Now i get another error. Its list index out of range. How can i fix it?
IndexError Traceback (most recent call last)
<ipython-input-17-c7bc734e5e35> in <module>()
1 if __name__ == '__main__':
----> 2 main()
<ipython-input-16-abe7438bdb9e> in main()
12 train_file = 'corpusfile.txt'
13 #load the corpus for the dataset
---> 14 token_len = loadCorpus(train_file, bi_dict, tri_dict, quad_dict, vocab_dict)
15
16 #estimate the lambdas for interpolation
<ipython-input-3-47bb9304cafa> in loadCorpus(file_path, bi_dict, tri_dict, quad_dict, vocab_dict)
97
98 #store the last few words for the next sentence pairing
---> 99 w1 = token[n -3]
100 w2 = token[n -2]
101 w3 = token[n -1]
IndexError: list index out of range