sebkur/wikiclean

A Java Wikipedia markup to plain text converter

WikiClean

WikiClean is a Java Wikipedia markup to plain text converter. It takes Wikipedia XML dumps with articles in wikimedia markup and generates clean plain text.

Why?

For text processing applications, we often need access to plain text, unadulterated by wikimedia markup. This is surprisingly non-trivial, as Wikipedia articles are full of complexities such as references, image captions, tables, infoboxes, etc., which are not useful for many applications.

Before setting out to write this package, I explored many of the Java alternatives for parsing Wikipedia pages described here and found none of them to be adequate for generating clean plain text. The primarily challenge is that most of these packages aspire to be complete Wikipedia parsers (e.g., for rendering), whereas WikiClean was designed with a much simpler goal — wiki markup to plain text conversion (nothing more, nothing less).

Usage

It's simple to use WikiClean:

WikiClean cleaner = new WikiClean.Builder().build();
String content = cleaner.clean(raw);

Where raw is the raw Wikpedia XML.

The builder allows you to specify a few options:

• withTitle to specify whether to prepend the article title in the plain text.
• withFooter to specify whether to keep the sections "See also", "Reference", "Further reading", and "External links".

By default, both options are set to false.

Also, use withLanguage to set the language. Currently, 17 are supported:

• English (default)
• Breton (Brezhoneg - br)
• Catalan (Català - ca)
• Chinese (中文 - zh)
• Dutch (Nederlands - nl)
• Esperanto (Esperanto - eo)
• French (Français - fr)
• Galician (Galego - gl)
• German (Deutsch - de)
• Greek (Ελληνικά - el)
• Italian (Italiano - it)
• Kannada (ಕನ್ನಡ - kn)
• Polish (Polski - pl)
• Portuguese (Português - pt)
• Russian (Русский - ru)
• Spanish (Español - es)
• Ukrainian (Українська - uk)

The corresponding classes are in org.wikiclean.languages.

Contributions for providing additional language support welcome!

Putting everything together, the default builder is equivalent to:

WikiClean cleaner =
    new WikiClean.Builder()
        .withLanguage(new English())
        .withTitle(false)
        .withFooter(false).build();
String content = cleaner.clean(raw);

Sample command-line invocation to read a Wikipedia dump and output plain text:

./scripts/wikipedia-articles-dump -input enwiki-20161220-pages-articles.xml.bz2 | less

Maven Artifacts

Latest releases of Maven artifacts are available at Maven Central.

License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0