Comments (2)
If you have reliable information about paragraphs there's no reason not to model paragraphs other than the increase in computation time. Chapters are great as well. Otherwise every 1000 words would work -- but the choice is arbitrary.
from tatom.
All valid choices, I guess. Ideally, I think we would split on borders between "scenes", the assumption(s) being that scenes form a meaningful unit, that they may have just the right size (although they may be very unequal in length), and that it makes sense to keep such units intact for best results from topic modeling. However, we usually don't have any information about scene boundaries.
So the next larger unit we tend to have information about is the chapter. And the next smaller unit we tend to have informatin about is the paragraph. However, paragraphs in novels are sometimes extremely short, if you define their border by a newline. For example, if there is an extended dialogue, each statement by a person will be one paragraph. So I think the best solution we currently have is to use something like "around n words (maybe 1000, or 2000), but cutting only on paragraph boundaries".
What would be really interesting is a study comparing results of topic modeling for the same texts but using different splitting strategies. Just to see whether it even makes a big difference.
from tatom.
Related Issues (19)
- Clarify target audience / insert clarifications of underlying concepts
- Add discussion of pandas (in lieu of numpy?) HOT 4
- build in some sort of sidebar
- try getting jquery fancybox working for images
- mention nltk's MALLET wrapper
- Installation of sklearn for Python3 on Linux HOT 2
- Build TAToM on Ubuntu 14.04 with Python 3.4 HOT 1
- Bayesian Group Comparison Typos HOT 1
- Installing Python packages on Debian-based systems HOT 1
- Webpage does not display correctly in Linux (chromium-browser and firefox) HOT 1
- UnicodeDecodeError HOT 1
- SyntaxError: invalid syntax on Topic Modeling script HOT 2
- Using PunktWordTokenizer
- Page not found https://de.dariah.eu/tatom/ HOT 4
- IndexError - Topic Modeling with Mallet Tutorial HOT 1
- syntax error HOT 2
- Is this a typo? HOT 1
- Host this material?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tatom.