malcolmgreaves/fact-extractor

Fact Extraction from Wikipedia Text

Fact Extractor

Fact Extraction from Wikipedia Text

Intro

The DBpedia Extraction Framework is pretty much mature when dealing with Wikipedia semi-structured content like infoboxes, links and categories.
However, unstructured content (typically text) plays the most crucial role, due to the amount of knowledge it can deliver, and few efforts have been carried out to extract structured data out of it.
For instance, given the Germany Football Team article, we want to extract a set of meaningful facts and structure them in machine-readable statements.
The following sentence:

In Euro 1992, Germany reached the final, but lost 0–2 to Denmark

would produce statements like:

<Germany, defeat, Denmark>
<defeat, score, 0–2>
<defeat, winner, Denmark>
<defeat, competition, Euro 1992>

Have you already?

Introduced yourself in the DBpedia GSoC mailing list? If not, please do so!

High-level Workflow + Related Code

INPUT = Wikipedia corpus (e.g., the latest Italian chapter)

1. Verb Extraction
2. Extract raw text from Wikipedia dump, forked from this repo
3. Extract a sub-corpus based on Wiki IDs
4. Verb Extraction
5. Verb Ranking
6. Build a frequency dictionary of lemmas
7. TF/IDF-based token ranking, using the TF/IDF module forked from this repo
8. Lemma ranking based on the token-to-lemma map
9. Training Set Creation
10. Build CrowdFlower input spreadsheet
11. Frame Classifier Training
12. Translate CrowdFlower results into training data format
13. Train classifier
14. Frame Extraction

Requirements

• TreeTagger
• libsvm
• csplit from GNU coreutils 8.23 or later

Warm-up Tasks

See the issues.

Development Policy

Committers should follow the standard team development practices:

1. Start working on a task
2. Branch out of master
3. Commit frequently with clear messages
4. Make a pull request

Coding Style

Pull requests not complying to these guidelines will be ignored.

• Use 4 spaces (soft tab) for indentation;
• Naming conventions
  • use an underscore as a word separator (files, variables, functions);
  • constants are UPPERCASE;
  • anything else is lowercase.
• Use 2 empty lines to separate functions;
• Write docstrings according to PEP 287, with a special attention to field lists. IDEs like PyCharm will do the job.

References

• FrameNet: A Knowledge Base for Natural Language Processing
• Outsourcing FrameNet to the Crowd
• Frame Semantics Annotation Made Easy with DBpedia

License

The source code is under the terms of the GNU General Public License, version 2.