Eidos is an open-domain machine reading system designed by the Computational Language Understanding (CLU) Lab at University of Arizona for the World Modelers DARPA program. Eidos uses a cascade of Odin grammars to extract causal events from free text.
Currently we extract entities such as "food insecurity" (and
increases
/decreases
/quantifications
of those entities) and directed causal
events that occur between entities such as "food insecurity causes increased
migration". In the near future we plan to expand this to also extract
correlation, same-as
, and is-a
relations.
Contents
This is a standard sbt project, so use the usual
commands, e.g., sbt compile
to compile or sbt assembly
to create a jar file.
sbt runMain
can be used to run some of the example applications directly, as
described below. To access Eidos from Java, add the assembled jar file(s) under
target/
to your $CLASSPATH. A file like eidos-assembly-0.1.6-SNAPSHOT.jar
may suffice, depending on the build. If necessary, see build.sbt
for a list
of runtime dependencies.
The Eidos system is designed to be used in several ways:
The scala API can produce three distinct output formats:
- a pretty display
- a JSON-LD export of the causal graph extracted from the text
- a JSON serialization (in case you want to later load all of the mentions, including mentions that are not part of the causal graph)
(see src/main/scala/org/clulab/wm/eidos/apps/examples/ExtractFromText.scala
)
import org.clulab.wm.eidos.EidosSystem
import org.clulab.wm.eidos.utils.DisplayUtils.displayMention
val text = "Water trucking has decreased due to the cost of fuel."
// Initialize the reader
val reader = new EidosSystem()
// Extract the mentions
val annotatedDocument = reader.extractFromText(text)
// Display in a pretty way
annotatedDocument.odinMentions.foreach(displayMention)
This produces the following output (mentions may appear in different order):
List(NounPhrase, Entity) => Water trucking
------------------------------
Rule => simple-np++Decrease_ported_syntax_2_verb
Type => TextBoundMention
------------------------------
NounPhrase, Entity => Water trucking
* Attachments: Decrease(decreased,None)
------------------------------
List(NounPhrase, Entity) => cost of fuel
------------------------------
Rule => simple-np
Type => TextBoundMention
------------------------------
NounPhrase, Entity => cost of fuel
------------------------------
List(Causal, DirectedRelation, EntityLinker, Event) => Water trucking has decreased due to the cost of fuel
------------------------------
Rule => dueToSyntax2-Causal
Type => EventMention
------------------------------
trigger => due
cause (NounPhrase, Entity) => cost of fuel
effect (NounPhrase, Entity) => Water trucking
* Attachments: Decrease(decreased,None)
------------------------------
(For information about the JSON-LD export format, please look here.)
import scala.collection.Seq
import org.clulab.serialization.json.stringify
import org.clulab.wm.eidos.EidosSystem
import org.clulab.wm.eidos.serialization.json.JLDCorpus
val text = "Water trucking has decreased due to the cost of fuel."
// Initialize the reader
val reader = new EidosSystem()
// Extract the mentions
val annotatedDocument = reader.extractFromText(text)
// Export to JSON-LD
val corpus = new JLDCorpus(Seq(annotatedDocument), reader)
val mentionsJSONLD = corpus.serialize()
println(stringify(mentionsJSONLD, pretty = true))
This produces the JSON-LD output shown here.
import org.clulab.serialization.json.stringify
import org.clulab.wm.eidos.EidosSystem
import org.clulab.wm.eidos.serialization.json.WMJSONSerializer
val text = "Water trucking has decreased due to the cost of fuel."
// Initialize the reader
val reader = new EidosSystem()
// Extract the mentions
val annotatedDocument = reader.extractFromText(text)
// Or... optionally serialize to regular JSON
// (e.g., if you want to later reload the mentions for post-processing)
val mentionsJSON = WMJSONSerializer.jsonAST(annotatedDocument.odinMentions)
println(stringify(mentionsJSON, pretty = true))
This produces the JSON serialization here (mentions may appear in different order):
sbt "runMain org.clulab.wm.eidos.apps.ExtractFromDirectory /path/to/input/directory /path/to/output/directory"
Files in the input directory should end with txt
and the extracted mentions
from each file will be saved in corresponding JSON-LD files.
Note: You cannot use tildes (~
) in the invocation in lieu of the home
directory.
The EidosShell is an interactive shell for testing the output of Eidos. To run it, do
./shell
To run the webapp version of EidosShell locally, do:
sbt webapp/run
and then navigate to localhost:9000
in a web browser.
Eidos reading output can be visualized using INDRA and Jupyter notebooks. See below for an example.
Events extracted using Eidos can be converted to INDRA Influence statements, which are bespoke data structures designed for modeling causal networks.
Example usage:
>>> from indra.sources import eidos
>>> ep = eidos.process_text("Water trucking has decreased due to the cost of fuel.")
>>> ep.statements
[Influence(cost of fuel(), Water trucking(negative))]
Delphi is a framework built on top of INDRA that assembles causal fragments extracted by Eidos into a causal analysis graph. This causal analysis graph is then converted to a dynamic Bayes network and used to make probabilistic predictions.
While we will soon be licensed as Apache, currently one dependency has a GPL licence. This will be removed very soon and the license will be updated.
If you are working on this project, you may be interested in additional materials stored in the cloud. Access may be limited by permission settings. Other documents are included in the /doc directory of the repository.
Of particular interest for those involved with grounding are two very large files of vectors that are used for the Word2Vec algorithm. They are
To use these files, download one or both and place them in the project's
src/main/resources/org/clulab/wm/eidos/w2v
directory. The first file can be used as is, but
the second file needs to be extracted before use with a command like
tar xvzf glove.840B.300d.txt.tgz
.) Indicate to Eidos
that they should be used by setting useW2V = true
in
src/main/resources/eidos.conf
. Only one can be used at a time, so make sure the
appropriate one of these lines is uncommented, without the //
.
wordToVecPath = "/org/clulab/wm/eidos/w2v/vectors.txt"
wordToVecPath = "/org/clulab/wm/eidos/w2v/glove.840B.300d.vectors.txt"
Processing the files can consume multiple minutes of time, so if you want to run Eidos more than a couple of times with the vectors, then it's useful to cache a processed version of them along with the ontologies they work with. This can be accomplished with the command
sbt "runMain org.clulab.wm.eidos.apps.CacheOntologies"
This will by default write serialized versions of the known ontologies and configured
vector file to ./cache/
. To use the cached copies, set useCachedOntologies = true
in src/main/resources/eidos.conf
.
The default size of the memory allocation pool for the JVM is 1/4 of your physical memory, but Eidos may require more RAM than that. It is currently being developed and tested with a 6GB limit.
For those using sbt
, the file .jvmopts
is included with the source code to
arrange for more memory. No other changes should be necessary.
IDEs and other development tools are generally unaware of .jvmopts
, but can be
configured via an environment variable instead.
JAVA_TOOL_OPTIONS=-Xmx6g
Other situations may require a more general setting.
_JAVA_OPTIONS=-Xmx6g
The procedure for defining these variables is dependent on operating system and shell.