A semi-unsupervised language independent morphological analyzer useful for stemming unknown language text, or getting a rough estimate of possible parses for morphemes in a word. Uses compression, maximum entropy and fieldlinguistics.

$ npm install --save ilanguage

More examples

var ILanguage = require('ilanguage').ILanguage;
var lang = new ILanguage(); 
var textToTest = {
  orthography: "this will not have any stop words or morphemes, vulgar words or unrepresentative words like banana",
  nonContentWordsArray: "not any",
  userSpecifiedNonContentWords: true,
  userRemovedWordsForThisDocumentArray: ['banana'],
  userRemovedWordsForAllDocumentsArray: ['vulgar'],
  // morphemes: /(^un|^pre|s$|ed$|ing$)/,
  morphemesArray: ["un-", " pre-", " -s", " -ed", " -ing"]
};
NonContentWords.processNonContentWords(textToTest);
expect(NonContentWords.filterText(textToTest).filteredText)
  .toEqual('thi will  have  stop word or morpheme  word or representative word like ');

• Gina Chiodo (U Delaware, Concordia)
• Theresa Deering (McGill, Aquafadas)
• Josh Horner (Amilia)
• Mathieu Legault (HEC, UQAM, Pivot88)
• Hisako Noguchi (Concordia)
• Tobin Skinner (McGill, Amilia)

• M.E. Cathcart 2012 (U Delaware)
• Alexandra Marquis 2013-2014 (UQAM, U de Montréal)
• Joel Dunham 2014-2015 (UBC)

• Siddartha Kattoju Summer 2011 (Concordia, Electrical Engineering)
• Curtis Mesher Fall 2011, Spring 2012 (Concordia, Theoretical Linguistics)
• Diana Olepeka Fall 2011 (Concordia, Theoretical Linguistics)
• Elise McClay Fall 2012, Spring 2013 (McGill, Field Linguistics)
• Bahar Sateli Spring 2012 (Concordia, Software Engineering)
• Yuliya Manyakina Summer 2012 (McGill, Field Linguistics)
• Xianli Sun Spring 2013 (Miami University, Software Engineering)
• Louisa Bielig Summer 2013, Fall 2013, Summer 2015 (McGill, Theoretical Linguistics)
• Dominique Bédard Fall 2013, Spring 2014 (U de Montréal, Speech Language Pathology)
• Alexandre Herbay Spring 2014 (U de Montréal, Psycho-linguistics & Toulouse III, Computer Science)
• Veronica Cook-Vilbrin Summer 2015 (Norwich University, Psychology)

• v1.0 April 16 2009 - Initial implementation in bash and perl
• v2.0 Jul 3 2010 - Implementation in C++
• v3.0 April 30 2011 - Implementation in Groovy
• v4.0 July 20 2012 - Implementation in JavaScript Map Reduce
• v4.1 Nov 29 2013 - Added more high level functions for gloss lookup
• v5.0 Jan 9 2014 - Implementation in CommonJS

This project is released under the Apache 2.0 license, which is an very non-restrictive open source license which basically says you can adapt the code to any use you see fit.

Sublime will manage this for you if you format (CMD+SHIFT+P, format) your code when you save. You can refer to .editorconfig and .jshintrc for specific options.

You can open the test/SpecRunner.html in an actual browser to run the unit test file(s) or breakpoint the code.

In general, you should always ensure that you have the latest Node.js and npm installed. On Mac you can do this by

brew update
brew upgrade node

Test that Gulp is installed by running gulp --version. If the command isn't found, run npm install -g gulp. For more information about installing the tools, see the getting started with Gulp guide.

1. Fork the repo.
2. Clone the repo to your computer.
3. Run npm install to install all build dependencies.
4. Run gulp to build this project.

Assuming that it looks something like this, you're ready to go.

Easy way

1. Signup for a GitHub account (GitHub is free for OpenSource)
2. Click on the "Fork" button to create your own copy.
3. Leave us a note in our issue tracker to tell us a bit about the bug/feature you want to work on.
4. You can follow the 4 GitHub Help Tutorials to install and use Git on your computer.
5. Feel free to ask us questions in our issue tracker, we're friendly and welcome Open Source newbies.
6. Edit the code on your computer, commit it referencing the issue #xx you created ($ git commit -m "fixes #xx i changed blah blah...") and push to your origin ($ git push origin master).
7. Click on the "Pull Request" button, and leave us a note about what you changed. We will look at your changes and help you bring them into the project!

Advanced way

1. Create a new branch for new fixes or features, this is easier to build a fix/feature specific pull request than if you work in your master branch directly.
2. Run gulp watch which will run the tests as you make changes.
3. Add failing tests for the change you want to make. Run gulp test to see the tests fail.
4. Fix stuff.
5. Look at the terminal output (assuming you ran gulp watch) to see if the tests pass. Repeat steps 2-4 until done.
6. Open test/SpecRunner.html unit test file(s) in actual browser(s) (Chrome Canary, Firefox, Safari) to ensure tests pass everywhere.
7. Update the documentation to reflect any changes.
8. Push to your fork and submit a pull request.