derekgreene / topic-model-tutorial Goto Github PK
View Code? Open in Web Editor NEWTutorial on topic models in Python with scikit-learn
Tutorial on topic models in Python with scikit-learn
I am trying to calculate the coherence value on each topic but each time when I run my code it generates different values.
It will be a great help if anyone can answer this.
Thank you in advance
def build_w2c(self, raw_documents):
docgen = TokenGenerator(raw_documents, self.stop_words)
new_list = []
for each in docgen.documents:
new_list.append(each.split(" "))
# print(new_list)
# Build the word2vec model
self.w2v_model = gensim.models.Word2Vec(size=500, min_count=0.0005, sg=1)
self.w2v_model.build_vocab(sentences=new_list)
return self.w2v_model
def get_descriptor(self, all_terms, H, topic_index, top):
# reverse sort the values to sort the indices
top_indices = np.argsort(H[topic_index, :])[::-1]
# now get the terms corresponding to the top-ranked indices
top_terms = []
for term_index in top_indices[0:top]:
top_terms.append(all_terms[term_index])
return top_terms
def get_coherence(self, k, terms, H):
k_values = []
term_rankings = []
coherences = []
dict = {}
for topic_index in range(1, k):
print(topic_index)
descriptor = self.get_descriptor(terms , H, topic_index, 10)
term_rankings.append(descriptor)
# Now calculate the coherence based on our Word2vec model
#coherence = self.calculate_coherence(term_rankings)
coherences.append(self.calculate_coherence(term_rankings))
print("K=%02d: Coherence=%.4f" % (topic_index, coherences[-1]))
k_values.append(topic_index)
dict[topic_index] = coherences[-1]
max_key = max(dict, key=dict.get)
return k_values, coherences, max_key
def calculate_coherence(self, term_rankings):
overall_coherence = 0.0
for topic_index in range(len(term_rankings)):
# check each pair of terms
pair_scores = []
for pair in combinations(term_rankings[topic_index], 2):
pair_scores.append(self.w2v_model.similarity(pair[0], pair[1]))
# get the mean for all pairs in this topic
topic_score = sum(pair_scores) / len(pair_scores)
overall_coherence += topic_score
# get the mean score across all topics
return overall_coherence / len(term_rankings)
here is my code that I have used in my project.
Output Required:
Each time when I run my code coherence should be the same.
if you can help me to resolve this approach it would be a great help
thank you so much.
Can you please give the reference of how the text data has been collected?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.