mojojolo/textteaser

TextTeaser is an automatic summarization algorithm.

TextTeaser

TextTeaser is an automatic summarization algorithm that combines the power of natural language processing and machine learning to produce good results.

TextTeaser is ported in Python

Requirements

SBT is needed for TextTeaser to run. Use of Scala IDE is recommended

Setup

$ git clone https://github.com/MojoJolo/textteaser.git
$ sbt compile
$ sbt eclipse // If using Eclipse.
$ sbt run

textteaser's Issues

The number of sentences in the summary can be set.

Right now, the number of sentences in all summary is 5 sentences. It will be good if it can be set by the user.

Different results running locally vs. textteaser.com

Hey all, I saw this software running on reddit.com as a summarizing bot and I thought I would give it a try. I've got it running locally (via SimpleREPL), though the summary I'm getting is different than the one I get when I use the textteaser.com website.

The only "options" I've run across that seem to make any difference are the Summarizer private variables _summarySize and _keywordsSize. What settings/setup do I need in order to match the output from textteaser.com? Or is it just a matter of the website having a larger/different corpus? If so, where can I find/download a copy of the corpus?

After running com.textteaser.summarizer.Main, where is the default URL?

I'm running the scala app on EC2 server. What's the default URL for accessing the API? Suppose my domain is mydomain.com, do I just direct all POST to the mydomain.com?

Doesn't seem to be getting any response. My server tells me textteaser is running.

URL download and cleanup

Don't see the code for downloading content from a url, removing the boilerplate etc... Can you point me to the section of the code that does that?

tutorial on adding additional language support

i see there's a couple binary corpus files, but i don't see any info on how these are generated and/or how to add additional language support.

Error running your program.

Hello Jolo,
I have got the following error while running your program.

sbt run 
[info] Loading project definition from /home/aims/textteaser/project
[info] Set current project to textteaser (in build file:/home/aims/textteaser/)
[warn] Scala version was updated by one of library dependencies:
[warn] 	* org.scala-lang:scala-compiler:2.11.0 -> 2.11.1
[warn] To force scalaVersion, add the following:
[warn] 	ivyScala := ivyScala.value map { _.copy(overrideScalaVersion = true) }
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn] 	* com.google.guava:guava:(17.0, 16.0.1) -> 18.0
[warn] Run 'evicted' to see detailed eviction warnings

Multiple main classes detected, select one to run:

 [1] com.textteaser.summarizer.SimpleREPL
 [2] com.textteaser.summarizer.Main

Enter number: 2

[info] Running com.textteaser.summarizer.Main 
12:17:50.856 [run-main-0] INFO   - Starting...
12:17:51.080 [run-main-0] INFO   - App is now runnning.
[error] (run-main-0) com.foursquare.rogue.RogueException: Mongo query on ConnectionIdentifier(lift) [db.keywords.count({ "blog" : ""})] failed after 10035 ms
com.foursquare.rogue.RogueException: Mongo query on ConnectionIdentifier(lift) [db.keywords.count({ "blog" : ""})] failed after 10035 ms
	at com.foursquare.rogue.MongoJavaDriverAdapter.runCommand(MongoJavaDriverAdapter.scala:32)
	at com.foursquare.rogue.MongoJavaDriverAdapter.count(MongoJavaDriverAdapter.scala:46)
	at com.foursquare.rogue.QueryExecutor$class.count(QueryExecutor.scala:31)
	at com.foursquare.rogue.LiftQueryExecutor.count(LiftQueryExecutor.scala:36)
	at com.foursquare.rogue.ExecutableQuery.count(ExecutableQuery.scala:20)
	at com.textteaser.summarizer.MongoKeywordService.getBlogCount(MongoKeywordService.scala:11)
	at com.textteaser.summarizer.Summarizer$$anonfun$getTopKeywords$1.apply(Summarizer.scala:45)
	at com.textteaser.summarizer.Summarizer$$anonfun$getTopKeywords$1.apply(Summarizer.scala:44)
	at scala.collection.immutable.List.map(List.scala:272)
	at com.textteaser.summarizer.Summarizer.getTopKeywords(Summarizer.scala:44)
	at com.textteaser.summarizer.Summarizer.summarize(Summarizer.scala:33)
	at com.textteaser.summarizer.Main$.delayedEndpoint$com$textteaser$summarizer$Main$1(Main.scala:33)
	at com.textteaser.summarizer.Main$delayedInit$body.apply(Main.scala:13)
	at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
	at scala.App$class.main(App.scala:76)
	at com.textteaser.summarizer.Main$.main(Main.scala:13)
	at com.textteaser.summarizer.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
Caused by: com.mongodb.MongoTimeoutException: Timed out while waiting to connect after 762 ms
	at com.mongodb.BaseCluster.getDescription(BaseCluster.java:114)
	at com.mongodb.DBTCPConnector.getClusterDescription(DBTCPConnector.java:396)
	at com.mongodb.DBTCPConnector.getType(DBTCPConnector.java:569)
	at com.mongodb.DBTCPConnector.getReplicaSetStatus(DBTCPConnector.java:364)
	at com.mongodb.Mongo.getReplicaSetStatus(Mongo.java:465)
	at com.mongodb.DB.getCommandReadPreference(DB.java:86)
	at com.mongodb.DB.command(DB.java:314)
	at com.mongodb.DB.command(DB.java:296)
	at com.foursquare.rogue.MongoJavaDriverAdapter$$anonfun$count$1.apply$mcJ$sp(MongoJavaDriverAdapter.scala:59)
	at com.foursquare.rogue.MongoJavaDriverAdapter$$anonfun$count$1.apply(MongoJavaDriverAdapter.scala:46)
	at com.foursquare.rogue.MongoJavaDriverAdapter$$anonfun$count$1.apply(MongoJavaDriverAdapter.scala:46)
	at com.foursquare.rogue.QueryHelpers$DefaultQueryLogger.onExecuteQuery(QueryHelpers.scala:43)
	at com.foursquare.rogue.MongoJavaDriverAdapter.runCommand(MongoJavaDriverAdapter.scala:29)
	at com.foursquare.rogue.MongoJavaDriverAdapter.count(MongoJavaDriverAdapter.scala:46)
	at com.foursquare.rogue.QueryExecutor$class.count(QueryExecutor.scala:31)
	at com.foursquare.rogue.LiftQueryExecutor.count(LiftQueryExecutor.scala:36)
	at com.foursquare.rogue.ExecutableQuery.count(ExecutableQuery.scala:20)
	at com.textteaser.summarizer.MongoKeywordService.getBlogCount(MongoKeywordService.scala:11)
	at com.textteaser.summarizer.Summarizer$$anonfun$getTopKeywords$1.apply(Summarizer.scala:45)
	at com.textteaser.summarizer.Summarizer$$anonfun$getTopKeywords$1.apply(Summarizer.scala:44)
	at scala.collection.immutable.List.map(List.scala:272)
	at com.textteaser.summarizer.Summarizer.getTopKeywords(Summarizer.scala:44)
	at com.textteaser.summarizer.Summarizer.summarize(Summarizer.scala:33)
	at com.textteaser.summarizer.Main$.delayedEndpoint$com$textteaser$summarizer$Main$1(Main.scala:33)
	at com.textteaser.summarizer.Main$delayedInit$body.apply(Main.scala:13)
	at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.App$$anonfun$main$1.apply(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
	at scala.App$class.main(App.scala:76)
	at com.textteaser.summarizer.Main$.main(Main.scala:13)
	at com.textteaser.summarizer.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
	at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 52 s, completed 24 Jul, 2017 12:18:01 PM

Kindly help me. It is a bit urgent

Prescription from a pharmacy Quran guidance in the upbringing of children

4– The prescription of guidance in bringing up children

Do you know O believers that Allah has made a full course for the believers for bringing up their children? Either they are in the nursery, primary, preparatory, secondary or university stage? If they are young, there is a complete course in Sorat Lukman, Allah says:
"O my son! Join not in worship others with Allah. Verily! Joining others in worship with Allah is a great Zulm (wrong) indeed." (Surat Lukman, verse 13)
It begins by training the child to the oneness of Allah and then orders him:
"O my son! Aqim-As-Salat (perform prayers), enjoin (people) for Al-Ma'ruf (all that is good), and forbid (people) from Al-Munkar (i.e. all that is evil and bad), and bear with patience whatever befall you. Verily! These are some of the important commandments ordered by Allah with no exemption." (Surat Lukman, verse 17)
His duty towards his God is to perform prayers, to order Al-Ma'ruf and forbid Al-Munkar, his duty from his heart towards whatever befalls on him is to: "And bear with patience whatever befalls you." So that he won't be scared terrified or seduced of what befalls on him from the heavens and that which can be removed by supplication to Allah Who sent it, then it orders him towards people to:
"And turn not your face away from men with pride, nor walk in insolence through the earth. Verily, Allah likes not each arrogant boaster. And be moderate (or show no insolence) in your walking, and lower your voice. Verily, the harshest of all voices is the voice (braying) of the ass." (Surat Lukman, verses 18, 19)
By Allah, if you see a child upon these divine principles and Koranic instructions, by Allah he will be like the noble angels and he will be as the prophet (May the blessings and peace of Allah be upon him) said narrating after his God:
"O repentant young man, to Me you are as some of My angels."[1]
So he is like the angels of Allah. When he reaches the age of youth, Allah made a course for him in Surat Al-Kahf as he said:
"Truly! They were young men who believed in their Lord (Allah), and We increased them in guidance." (Surat Al-Kahf, verse 13)
He (Allah) detailed the noble course which:

• Protects the youths from temptation and preserves them from evils.
• Made them always cling to the right.
• Made them carry on the Sunna of the prophet and the religion of Allah.
• Made them apply the way of the Koran.
• Made them to be dutiful to their parents and always obedient to them.

[1] The prophet (May the blessings and peace of Allah be upon him) said: "Allah likes nothing better than a repentant youth." Al­Daylamy by Anas, Kanz Al­Om"mal.

http://www.fawzyabuzeid.com/en/table_books.php?name=Be%20Quran%20Walking%20Among%20People&id=48&cat=2

computation in Summarizer.dbs method is wrong

Original:
(r._1._1 * r._2._2) / Math.pow(r._1._2 - r._2._2, 2)

Which should be:
(r._1._1 * r._2._1) / Math.pow(r._1._2 - r._2._2, 2)

Which formula is this ? tell me why this formula are we using ?

when using computscore method that time you are calculating

val totalScore = (titleFeature * 1.5 + keywordFrequency * 2.0 + sentenceLength * 0.5 + sentencePosition * 1.0) / 4.0

why are you multiplying title feature with 1.5 and keywordfrequency

if you have any research paper regarding this please tell me ?

Is summarization algorithm available ?

Is the textteaser summarization algorithm available to read ? I've used it for some test cases and I'm interested to discover why it produces certain summaries.

Run TextTeaser in Eclipse

hi, how do I Teaser Text in eclipse? ubuntu shell everything works perfectly, now I want to work on the project in eclipse and I can not execute it

Address is invalid on local machine, or port is not valid on remote machine

Getting this error when I try to run after a successful compile:

at java.lang.reflect.Method.invoke(Unknown Source)
Caused by: java.io.IOException: couldn't connect to [/127.0.0.1:0] bc:java.net.ConnectException: connect: Address is invalid on local machine, or port is not valid on remote machine
at com.mongodb.DBPort._open(DBPort.java:214)
at com.mongodb.DBPort.go(DBPort.java:107)
at com.mongodb.DBPort.call(DBPort.java:74)
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:286)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:257)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:310)
at com.mongodb.DB.command(DB.java:274)
at com.mongodb.DB.command(DB.java:256)
at com.mongodb.DB.command(DB.java:313)
at com.mongodb.DB.command(DB.java:211)
at com.mongodb.DB$AuthenticationCredentials.authenticate(DB.java:870)
at com.mongodb.DB.authenticateCommandHelper(DB.java:641)
at com.mongodb.DB.authenticate(DB.java:601)
at net.liftweb.mongodb.MongoDB$.defineDbAuth(Mongo.scala:121)
at com.textteaser.summarizer.Main$delayedInit$body.apply(Main.scala:24)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at com.textteaser.summarizer.Main$.main(Main.scala:13)
at com.textteaser.summarizer.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
error Nonzero exit code: 1
[error] Total time: 19 s, completed Oct 12, 2013 7:52:45 PM

C:\Users\gratitude\textteaser>cd C:/xampp/htdocs

C:\xampp\htdocs>cd textteaser

C:\xampp\htdocs\textteaser>sbt compile
[info] Loading project definition from C:\xampp\htdocs\textteaser\project
[info] Updating {file:/C:/xampp/htdocs/textteaser/project/}default-0f03be...
[info] Resolving com.typesafe.sbteclipse#sbteclipse-plugin;2.2.0 ...
[info] Resolving com.typesafe.sbteclipse#sbteclipse-core;2.2.0 ...
[info] Resolving org.scalaz#scalaz-core_2.9.2;6.0.4 ...
[info] Resolving org.scala-lang#scala-library;2.9.2 ...
[info] Resolving com.typesafe.sbt#sbt-start-script;0.9.0 ...
[info] Resolving org.scala-sbt#sbt;0.12.0 ...
[info] Resolving org.scala-sbt#main;0.12.0 ...
[info] Resolving org.scala-sbt#actions;0.12.0 ...
[info] Resolving org.scala-sbt#classpath;0.12.0 ...
[info] Resolving org.scala-sbt#launcher-interface;0.12.0 ...
[info] Resolving org.scala-sbt#interface;0.12.0 ...
[info] Resolving org.scala-sbt#io;0.12.0 ...
[info] Resolving org.scala-sbt#control;0.12.0 ...
[info] Resolving org.scala-lang#scala-compiler;2.9.2 ...
[info] Resolving org.scala-sbt#completion;0.12.0 ...
[info] Resolving org.scala-sbt#collections;0.12.0 ...
[info] Resolving jline#jline;1.0 ...
[info] Resolving org.scala-sbt#api;0.12.0 ...
[info] Resolving org.scala-sbt#compiler-integration;0.12.0 ...
[info] Resolving org.scala-sbt#incremental-compiler;0.12.0 ...
[info] Resolving org.scala-sbt#logging;0.12.0 ...
[info] Resolving org.scala-sbt#process;0.12.0 ...
[info] Resolving org.scala-sbt#compile;0.12.0 ...
[info] Resolving org.scala-sbt#persist;0.12.0 ...
[info] Resolving org.scala-tools.sbinary#sbinary_2.9.0;0.4.0 ...
[info] Resolving org.scala-sbt#classfile;0.12.0 ...
[info] Resolving org.scala-sbt#compiler-ivy-integration;0.12.0 ...
[info] Resolving org.scala-sbt#ivy;0.12.0 ...
[info] Resolving org.apache.ivy#ivy;2.3.0-rc1 ...
[info] Resolving com.jcraft#jsch;0.1.46 ...
[info] Resolving commons-httpclient#commons-httpclient;3.1 ...
[info] Resolving commons-logging#commons-logging;1.0.4 ...
[info] Resolving commons-codec#commons-codec;1.2 ...
[info] Resolving org.scala-sbt#run;0.12.0 ...
[info] Resolving org.scala-sbt#task-system;0.12.0 ...
[info] Resolving org.scala-sbt#tasks;0.12.0 ...
[info] Resolving org.scala-sbt#tracking;0.12.0 ...
[info] Resolving org.scala-sbt#cache;0.12.0 ...
[info] Resolving org.scala-sbt#testing;0.12.0 ...
[info] Resolving org.scala-sbt#test-agent;0.12.0 ...
[info] Resolving org.scala-tools.testing#test-interface;0.5 ...
[info] Resolving org.scala-sbt#command;0.12.0 ...
[info] Resolving org.scala-sbt#compiler-interface;0.12.0 ...
[info] Resolving org.scala-sbt#precompiled-2_8_2;0.12.0 ...
[info] Resolving org.scala-sbt#precompiled-2_10_0-m4;0.12.0 ...
[info] Done updating.
[info] Set current project to textteaser (in build file:/C:/xampp/htdocs/textteaser/)
[info] Updating {file:/C:/xampp/htdocs/textteaser/}default-fcdec2...
[info] Resolving org.scala-lang#scala-library;2.10.1 ...
[info] Resolving com.google.code.gson#gson;2.2.4 ...
[info] Resolving org.apache.opennlp#opennlp-tools;1.5.2-incubating ...
[info] Resolving org.apache.opennlp#opennlp-maxent;3.0.2-incubating ...
[info] Resolving jwnl#jwnl;1.3.3 ...
[info] Resolving net.codingwell#scala-guice_2.10;3.0.2 ...
[info] Resolving com.google.inject#guice;3.0 ...
[info] Resolving javax.inject#javax.inject;1 ...
[info] Resolving aopalliance#aopalliance;1.0 ...
[info] Resolving org.sonatype.sisu.inject#cglib;2.2.1-v20090111 ...
[info] Resolving asm#asm;3.1 ...
[info] Resolving com.google.inject.extensions#guice-multibindings;3.0 ...
[info] Resolving com.google.guava#guava;11.0.1 ...
[info] Resolving com.google.code.findbugs#jsr305;1.3.9 ...
[info] Resolving org.json4s#json4s-native_2.10;3.2.4 ...
[info] Resolving org.json4s#json4s-core_2.10;3.2.4 ...
[info] Resolving org.json4s#json4s-ast_2.10;3.2.4 ...
[info] Resolving com.thoughtworks.paranamer#paranamer;2.5.2 ...
[info] Resolving org.scala-lang#scalap;2.10.0 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.1 ...
[info] Resolving org.scala-lang#scala-reflect;2.10.1 ...
[info] Resolving com.foursquare#rogue-field_2.10;2.2.0 ...
[info] Resolving com.foursquare#rogue-core_2.10;2.2.0 ...
[info] Resolving com.foursquare#rogue-lift_2.10;2.2.0 ...
[info] Resolving com.foursquare#rogue-index_2.10;2.2.0 ...
[info] Resolving net.liftweb#lift-mongodb-record_2.10;2.5.1 ...
[info] Resolving net.liftweb#lift-record_2.10;2.5.1 ...
[info] Resolving net.liftweb#lift-proto_2.10;2.5.1 ...
[info] Resolving net.liftweb#lift-webkit_2.10;2.5.1 ...
[info] Resolving net.liftweb#lift-util_2.10;2.5.1 ...
[info] Resolving net.liftweb#lift-actor_2.10;2.5.1 ...
[info] Resolving net.liftweb#lift-common_2.10;2.5.1 ...
[info] Resolving net.liftweb#lift-json_2.10;2.5.1 ...
[info] Resolving joda-time#joda-time;2.1 ...
[info] Resolving org.joda#joda-convert;1.2 ...
[info] Resolving commons-codec#commons-codec;1.6 ...
[info] Resolving javax.mail#mail;1.4.4 ...
[info] Resolving javax.activation#activation;1.1 ...
[info] Resolving nu.validator.htmlparser#htmlparser;1.4 ...
[info] Resolving commons-fileupload#commons-fileupload;1.2.2 ...
[info] Resolving org.scala-lang#scala-actors;2.10.0 ...
[info] Resolving net.liftweb#lift-mongodb_2.10;2.5.1 ...
[info] Resolving net.liftweb#lift-json-ext_2.10;2.5.1 ...
[info] Resolving org.mongodb#mongo-java-driver;2.10.1 ...
[info] Resolving org.slf4j#slf4j-api;1.7.5 ...
[info] Resolving ch.qos.logback#logback-classic;1.0.13 ...
[info] Resolving ch.qos.logback#logback-core;1.0.13 ...
[info] Done updating.
[info] Compiling 9 Scala sources to C:\xampp\htdocs\textteaser\target\scala-2.10\classes...
[success] Total time: 19 s, completed Oct 12, 2013 7:54:26 PM

C:\xampp\htdocs\textteaser>sbt run
[info] Loading project definition from C:\xampp\htdocs\textteaser\project
[info] Set current project to textteaser (in build file:/C:/xampp/htdocs/textteaser/)
[info] Running com.textteaser.summarizer.Main
19:54:44.845 [run-main] INFO - Starting...
Oct 12, 2013 7:54:45 PM com.mongodb.DBTCPConnector initDirectConnection
WARNING: Exception executing isMaster command on /127.0.0.1:0
java.io.IOException: couldn't connect to [/127.0.0.1:0] bc:java.net.ConnectException: connect: Address is invalid on local machine, or port is not valid on remote machine
at com.mongodb.DBPort._open(DBPort.java:214)
at com.mongodb.DBPort.go(DBPort.java:107)
at com.mongodb.DBPort.go(DBPort.java:88)
at com.mongodb.DBPort.findOne(DBPort.java:143)
at com.mongodb.DBPort.runCommand(DBPort.java:148)
at com.mongodb.DBTCPConnector.initDirectConnection(DBTCPConnector.java:548)
at com.mongodb.DBTCPConnector.isMongosConnection(DBTCPConnector.java:372)
at com.mongodb.Mongo.isMongosConnection(Mongo.java:627)
at com.mongodb.DB.wrapCommand(DB.java:294)
at com.mongodb.DB.command(DB.java:272)
at com.mongodb.DB.command(DB.java:256)
at com.mongodb.DB.command(DB.java:313)
at com.mongodb.DB.command(DB.java:211)
at com.mongodb.DB$AuthenticationCredentials.authenticate(DB.java:870)
at com.mongodb.DB.authenticateCommandHelper(DB.java:641)
at com.mongodb.DB.authenticate(DB.java:601)
at net.liftweb.mongodb.MongoDB$.defineDbAuth(Mongo.scala:121)
at com.textteaser.summarizer.Main$delayedInit$body.apply(Main.scala:24)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at com.textteaser.summarizer.Main$.main(Main.scala:13)
at com.textteaser.summarizer.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sbt.Run.invokeMain(Run.scala:68)
at sbt.Run.run0(Run.scala:61)
at sbt.Run.execute$1(Run.scala:50)
at sbt.Run$$anonfun$run$1.apply$mcV$sp(Run.scala:54)
at sbt.TrapExit$.executeMain$1(TrapExit.scala:33)
at sbt.TrapExit$$anon$1.run(TrapExit.scala:42)

Oct 12, 2013 7:54:45 PM com.mongodb.DBTCPConnector initDirectConnection
WARNING: Exception executing isMaster command on /127.0.0.1:0
java.io.IOException: couldn't connect to [/127.0.0.1:0] bc:java.net.ConnectException: connect: Address is invalid on local machine, or port is not valid on remote machine
at com.mongodb.DBPort._open(DBPort.java:214)
at com.mongodb.DBPort.go(DBPort.java:107)
at com.mongodb.DBPort.go(DBPort.java:88)
at com.mongodb.DBPort.findOne(DBPort.java:143)
at com.mongodb.DBPort.runCommand(DBPort.java:148)
at com.mongodb.DBTCPConnector.initDirectConnection(DBTCPConnector.java:548)
at com.mongodb.DBTCPConnector.checkMaster(DBTCPConnector.java:527)
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:277)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:257)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:310)
at com.mongodb.DB.command(DB.java:274)
at com.mongodb.DB.command(DB.java:256)
at com.mongodb.DB.command(DB.java:313)
at com.mongodb.DB.command(DB.java:211)
at com.mongodb.DB$AuthenticationCredentials.authenticate(DB.java:870)
at com.mongodb.DB.authenticateCommandHelper(DB.java:641)
at com.mongodb.DB.authenticate(DB.java:601)
at net.liftweb.mongodb.MongoDB$.defineDbAuth(Mongo.scala:121)
at com.textteaser.summarizer.Main$delayedInit$body.apply(Main.scala:24)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at com.textteaser.summarizer.Main$.main(Main.scala:13)
at com.textteaser.summarizer.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sbt.Run.invokeMain(Run.scala:68)
at sbt.Run.run0(Run.scala:61)
at sbt.Run.execute$1(Run.scala:50)
at sbt.Run$$anonfun$run$1.apply$mcV$sp(Run.scala:54)
at sbt.TrapExit$.executeMain$1(TrapExit.scala:33)
at sbt.TrapExit$$anon$1.run(TrapExit.scala:42)

Oct 12, 2013 7:54:45 PM com.mongodb.DBPortPool gotError
WARNING: emptying DBPortPool to /127.0.0.1:0 b/c of error
java.io.IOException: couldn't connect to [/127.0.0.1:0] bc:java.net.ConnectException: connect: Address is invalid on local machine, or port is not valid on remote machine
at com.mongodb.DBPort._open(DBPort.java:214)
at com.mongodb.DBPort.go(DBPort.java:107)
at com.mongodb.DBPort.call(DBPort.java:74)
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:286)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:257)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:310)
at com.mongodb.DB.command(DB.java:274)
at com.mongodb.DB.command(DB.java:256)
at com.mongodb.DB.command(DB.java:313)
at com.mongodb.DB.command(DB.java:211)
at com.mongodb.DB$AuthenticationCredentials.authenticate(DB.java:870)
at com.mongodb.DB.authenticateCommandHelper(DB.java:641)
at com.mongodb.DB.authenticate(DB.java:601)
at net.liftweb.mongodb.MongoDB$.defineDbAuth(Mongo.scala:121)
at com.textteaser.summarizer.Main$delayedInit$body.apply(Main.scala:24)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at com.textteaser.summarizer.Main$.main(Main.scala:13)
at com.textteaser.summarizer.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sbt.Run.invokeMain(Run.scala:68)
at sbt.Run.run0(Run.scala:61)
at sbt.Run.execute$1(Run.scala:50)
at sbt.Run$$anonfun$run$1.apply$mcV$sp(Run.scala:54)
at sbt.TrapExit$.executeMain$1(TrapExit.scala:33)
at sbt.TrapExit$$anon$1.run(TrapExit.scala:42)

error com.mongodb.MongoException$Network: can't call something : /127.0.0.1:0/
com.mongodb.MongoException$Network: can't call something : /127.0.0.1:0/
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:295)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:257)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:310)
at com.mongodb.DB.command(DB.java:274)
at com.mongodb.DB.command(DB.java:256)
at com.mongodb.DB.command(DB.java:313)
at com.mongodb.DB.command(DB.java:211)
at com.mongodb.DB$AuthenticationCredentials.authenticate(DB.java:870)
at com.mongodb.DB.authenticateCommandHelper(DB.java:641)
at com.mongodb.DB.authenticate(DB.java:601)
at net.liftweb.mongodb.MongoDB$.defineDbAuth(Mongo.scala:121)
at com.textteaser.summarizer.Main$delayedInit$body.apply(Main.scala:24)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at com.textteaser.summarizer.Main$.main(Main.scala:13)
at com.textteaser.summarizer.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
Caused by: java.io.IOException: couldn't connect to [/127.0.0.1:0] bc:java.net.ConnectException: connect: Address is invalid on local machine, or port is not valid on remote machine
at com.mongodb.DBPort._open(DBPort.java:214)
at com.mongodb.DBPort.go(DBPort.java:107)
at com.mongodb.DBPort.call(DBPort.java:74)
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:286)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:257)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:310)
at com.mongodb.DB.command(DB.java:274)
at com.mongodb.DB.command(DB.java:256)
at com.mongodb.DB.command(DB.java:313)
at com.mongodb.DB.command(DB.java:211)
at com.mongodb.DB$AuthenticationCredentials.authenticate(DB.java:870)
at com.mongodb.DB.authenticateCommandHelper(DB.java:641)
at com.mongodb.DB.authenticate(DB.java:601)
at net.liftweb.mongodb.MongoDB$.defineDbAuth(Mongo.scala:121)
at com.textteaser.summarizer.Main$delayedInit$body.apply(Main.scala:24)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at com.textteaser.summarizer.Main$.main(Main.scala:13)
at com.textteaser.summarizer.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
error Nonzero exit code: 1
[error] Total time: 2 s, completed Oct 12, 2013 7:54:45 PM

the fair state. This is the dream and the aim of mankind at every time.

"Allah made truth at Omar's tongue and heart."
This judge remained at his work for a year without a single case from anyone. The caliph wanted to give him his salary for the year, but Omar refused and said to Abu-Bakr: "People who believed in Allah, followed their prophet and made the Koran the judge among them, never need a judge to judge them." Because the judge is Allah and the dominant is the book of Allah.
When a man of them heard a verse from the book of Allah recited, he stops and says to himself: "This is the book of my Lord in which He never strays or forgets." He followed the book of Allah because it is the rescuer from all diseases, the cure for every illness and the assured healing for all problems. The prophet (May the blessings and peace of Allah be upon him) said:
"There will be temptation." They asked: "What is the way out O messenger of Allah?" He said: "The book of Allah, it has the news of people who proceeded you and people who will follow you and the judgments of what is amongst you, it is the unmistakable judgment not drollery or jesting, if an arrogant leaves it, Allah will shatter him, whoever seeks guidance away from it Allah strays him, it is the strong Rope of Allah, the wise remembrance and the straight way, the lusts never deviate by it, the tongues never ambiguous by it, scholars never be sated by it, it never wears out because of replying and there is no end to its marvels, when Jinn heard it they said: (Verily! we have heard a wonderful Recital (this Koran!) It guides to the Right Path, and we have believed therein.), whoever says it, he is truthful, whoever follows it, he will get rewards, whoever judges by it, he is fair and whoever calls for it, he will be guided to the straight way."
All the companions knew these meanings and made them their examples, our sir Abdullah Ibn Masood expressed this fact, he said: "We made the book of our God, messages from Allah to us, so we read and contemplated it at night then we do and fulfill it at day." I.e. fulfill what Allah orders in the Koran. If He said forgive, he forgives, if He said overlook (mistakes), he overlooks and if He said listen to so and so, he listens. He fulfills the Koran literally, everyone of them was a Koran walking among people following their prophet (May the blessings and peace of Allah be upon him) may Allah be satisfied with them.
You know when the prophet was gifted a doctor, he sent him back, because they followed the book of Allah and discovered the medicines that cures the diseases of man from the nearby medical plants, so they didn't need a doctor. Mrs. A'isha learned medicine from wise people who came to visit the messenger of Allah and prescribed medicines to him. When the prophet (May the blessings and peace of Allah be upon him) migrated from Mecca to Medina, they applied the instructions of the book of Allah all over their political, religious and social life. The result was that they made for mankind a perfect example for utopia (the virtuous city) and the fair state. This is the dream and the aim of mankind at every time.

http://www.fawzyabuzeid.com/en/table_books.php?name=Be%20Quran%20Walking%20Among%20People&id=48&cat=2

Is it possible to disable the use of MongoDB?

If category or blog hints/fields are never set or used?

Build a corpus for other languages.

TextTeaser uses OpenNLP to split sentences. And OpenNLP requires a corpus for it to split sentences properly.

Here's a list of languages with corpus from OpenNLP:
http://opennlp.sourceforge.net/models-1.5/

Limited language are supported by the sentence detector of OpenNLP. It's good if we can have other language supported too. E.g. Russian, Chinese, Japanese.

Here's an instruction to create a sentence detector corpus:
http://opennlp.apache.org/documentation/1.5.3/manual/opennlp.html#tools.sentdetect.training

Set proper encoding for the source files

sbt compile outputs the following on an Ubuntu 64bits

[info] Loading project definition from /home/mea/Development/sandbox/textteaser/project
[info] Set current project to textteaser (in build file:/home/mea/Development/sandbox/textteaser/)
[info] Compiling 9 Scala sources to /home/mea/Development/sandbox/textteaser/target/scala-2.10/classes...
[error] IO error while decoding /home/mea/Development/sandbox/textteaser/src/main/scala/com/textteaser/summarizer/Main.scala with UTF-8
[error] Please try specifying another one using the -encoding option
[error] one error found
error Compilation failed
[error] Total time: 3 s, completed Oct 12, 2013 11:46:31 AM

how to convert query into Mysql if we use Mysql ?

I want to convert this query def getBlogScore(word: String, blog: String): Long = Keyword.where(_.word eqs word).and(_.blog eqs blog) .fetch.map(_.score._1) .reduceLeftOption(_ + _).getOrElse(0)

into Mysql ?

How to do this?

Create a good documentation in Readme

Anyone?

How Did Allah Prepare The Greatest Doctor Of The Pharmacy Mohammed

How Did Allah Prepare The Greatest
Doctor Of The Pharmacy Mohammed
(May the blessings and peace of Allah be upon him)

In order not to be confused and to be assured of the cures of our Lord, we say by Allah's guidance:
Allah prepared the chosen beloved (May the blessings and peace of Allah be upon him) as he is the greatest doctor and the honored wise man for this unique mission, then He made him the perfect example and the practical live model. He (upon him the best pray and the perfect peace) was a Koran walking on land. When Mrs. Aa'Isha was asked about his manners she said:
"His manners were the Koran, don't you read the Koran, Allah says: And verily, you (O Muhammad) are on an exalted standard of character." [1]
Let's now see how Allah prepared his beloved by these beauties, perfections, cures, secrets and lights until he became this unique perfect example going among people.
Firstly:
Allah chose him, brought him up under His eye and taught him by Himself. He didn't have a teacher or an instructor, so that no one has grace or guardianship upon him or his nation, Allah said to him in the Koran assuring this unique honor:
"And (Allah) taught you that which you knew not. And Ever Great is the Grace of Allah unto you (O Muhammad)." (Surat Al-Nesaa, verse 113)
He (May the blessings and peace of Allah be upon him) said about that when our sir Aba-Bakr asked him:
"O messenger of Allah I knew princes, great men and kings, I didn't see your manners, who taught you O messenger of Allah?" He said:
"My lord brought me up so well." [2]
Allah made him wise in his words, an example in his behavior and deeds and perfect in his manners until he became so perfect in his manners, Allah praised him in His book e.g.:
"And verily, you (O Muhammad) are on an exalted standard of character." (Surat Al-Kalam, verse 4)

Allah informed us all that the perfect guidance in any matter concerning religion, this life and the hereafter, only comes by obeying him, He says:
"If you obey him, you shall be on the right guidance." (Surat An'Nour, verse 54)
If we obey him, we will be guided, why we look for solutions to our economic problems with nations that all their principles are personal principles made by men with desires and mistakes and leave the perfect tested healings with no mistakes, Allah says about them:
"It is only an Inspiration that is inspired." (Surat An'Najm, verse 4)
So we can say that Allah gave him luminous spirit to infer the needs of human societies, and spiritual transparent insight to look at the illnesses of souls, bodies and societies to put by the order and permission of Allah, a cure for every disease with perfect wisdom and high mercy by which Allah described his relation with his nation and set this in His Koran and said to us about that:
"For the believers (he is) full of pity, kind, and merciful." (Surat At'Tawba, verse 128)
This was the first preparation to him (May the blessings and peace of Allah be upon him) in the spiritual and moral side.
Secondly:
At the practical and executive side in Da'wa:

• Allah made him call to Him in Mecca for thirteen years.
• He faced unmatched opposition, persecution and annoying there.
• He was (May the blessings and peace of Allah be upon him) an example for the wise caller who calls to Allah:
  · By his words (May the blessings and peace of Allah be upon him) at all times.
  · By his deed at all deeds without exception.
  · By his status however he was.
  · By wisdom and fair preaching.
  · And at all circumstances, either can be realized or above imagination.

•                  He began his call as a single person, no one with him but Allah, no weapon but the Koran which was being revealed verses after verses.
  

•                  In few years, his call spread and followed by different nations.
  

  Thirdly:
  The third preparation is a special one:
  No human being acquires a similar one:
  "He studied making nations, building countries and societies." This needs not only tens of years but hundreds of years to be achieved, but by this special preparation by Allah to His messenger (May the blessings and peace of Allah be upon him), in few years he built a great nation led the whole world. This never happened in history like this except this time, Allah didn't want people to think that Islam is a religion of worship and dealings and has no concern with this life, He made His beloved migrates to Medina:

•                  To be the perfect example in establishing urban societies from his age to the end of this world.
  

•                  He established a society upon fearing of Allah and watching Him, he got rid of all the political, social, economic or other problems happened in urban societies, to be sure that Islam is a religion and a country, a religion for Allah and a country organizes the rights of individuals and groups and reform them to be good societies without problems, if they followed the guidance of the beloved and chosen of Allah (May the blessings and peace of Allah be upon him).
  

  So you see that through all these stages, statuses and developments, Allah made his life (May the blessings and peace of Allah be upon him) full of all different problems and diseases that can happen to mankind until the day of judgment, either for individuals (leaders or subordinates) or families and groups (relatives, enemies, friends, hypocrites, enviers, spiteful people, previous religion's people, different disbelievers, powerless, arrogant, the poor, the rich, ……) or small
  http://www.fawzyabuzeid.com/en/table_books.php?name=Be%20Quran%20Walking%20Among%20People&id=48&cat=2

[1] Saad Ibn Hesham Ibn Amer said: "I came to Aa'Isha and said: O Mother of believers, tell me the manners of the messenger of Allah," she said: "His manners were the Quran, don't you read the Quran, Allah says: And verily, you (O Muhammad) are on an exalted standard of character." Mosnad Imam Ahmed.
[2] Ibn Al-Samaany in Adab Al-Imlaa by Ibn Masood, Jamea Al-Ahadeeth Wal-Maraseel.

IO error while decoding /Scala/textteaser/src/main/scala/com/textteaser/summarizer/Main.scala with UTF-8

So I am following the steps. Clone the repo. Cd into it. Run sbt compile and then get these error messages. I am new to scala, not sure how to resolve. Any thoughts?

$ sbt --version
sbt launcher version 0.13.0

[error] IO error while decoding /Scala/textteaser/src/main/scala/com/textteaser/summarizer/Main.scala with UTF-8
[error] Please try specifying another one using the -encoding option
[error] one error found
error Compilation failed
[error] Total time: 5 s, completed 12-Oct-2013 14:57:11

Testing Textteaser

I think writing some at least basic tests should be a good idea considering ongoing development and changes, so I'd like to start with something, hence I'd like to know what testing framework do you suggest.
I suggest using ScalaTest / FunSuite as rather traditional and reliable tool.

Error with unresolved dependency github.com/mpeltonen/sbt-idea;1.6.0-SNAPSHOT

When trying to run sbt run or sbt eclipse, I get the following error:

[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: com.github.mpeltonen#sbt-idea;1.6.0-SNAPSHOT: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn]  Note: Some unresolved dependencies have extra attributes.  Check that these dependencies exist with the requested attributes.
[warn]      com.github.mpeltonen:sbt-idea:1.6.0-SNAPSHOT (sbtVersion=0.12, scalaVersion=2.9.2)
[warn]

...

[error] (*:update) sbt.ResolveException: unresolved dependency: com.github.mpeltonen#sbt-idea;1.6.0-SNAPSHOT: not found
Project loading failed

I'm on a Mac OS X 10.9.3 if that makes a difference. I've never coded in Java so eclipse is rather new to me.

Is there a workaround, or am I not doing something right?
Any help would be appreciated. Thanks!

Why is 'phillipines' in your stop words?

Phillipines and manilla are in the stop words?
https://github.com/MojoJolo/textteaser/blob/master/src/main/scala/com/textteaser/summarizer/Parser.scala

do you have something against these places in particular?

Give me a auto short making ai tools

knowing that true happiness comes from appreciatingOliver returned to collecting acorns,Grateful for his simple acorn treasures,he wished to undo it all.

Improve sentence splitter