Giter Site home page Giter Site logo

xiaolongn100 / nlp4go Goto Github PK

View Code? Open in Web Editor NEW

This project forked from korobool/nlp4go

0.0 1.0 0.0 8.55 MB

Basic libraries and utils for Natural Language Processing with GoLang

License: MIT License

Go 95.88% Python 3.47% Makefile 0.24% Shell 0.41%

nlp4go's Introduction

nlp4go

Join the chat at https://gitter.im/korobool/nlp4go Build Status Coverage Status

Note: This solution is not under development anymore. Further development was done under NDA and couldn't be published

The idea of nlp4go is to provide a fast go-lang based nlp toolkit for researchers and developers which provides the most commonly used features of NLTK and other NPL toolkits, but with production-ready computational performance. Things that can be executed in parallel mode (like POS tagging for independent sentences) should be processed in go-routines in parallel to utilize CPU cores efficiently.

Currently supported languages

  • English
  • Russian < Planned

General plan for implementation

  • Tokenizer(s)
  • Tree bank
  • regex
  • split
  • investigate alternatives
  • POS tagger(s)
  • Percepton
  • String abstraction to imporove performance on unicode
  • Compatibility with regex
  • O(1) len() operation for unicode strings
  • Slises in bytes and characters
  • NER support
  • Parsing
  • We need a complete rules set for syntax parsing
  • Syntax Parsing // ShiftReudce
  • Dependency Parsing // Port Hannibal's Cython version
  • WordNet interface

Repository structure should be idiomatically similar to following tree:

legacy  // to be removed later
core
    strings.go
    exregex.go
ml
    perceptron.go 
    ...
tokenize
    wordsplit.go
    wordregex.go
    sentencesplit.go
    sentencesregex.go
    ...
tagg
    perceptron-pos
parse
    syntax
    dependency
    ...
utils
   train_pos_tagger.go
   read_ontonotes.go

Train model for perceptron based POS tagger

go run tagger_train.go -corpus /home/user/ontonotes -model test-model.go

Run STDIN tagger

go run tagger_tag.go -model test-model.go

We cannot share ontonotes, but you can use your own training data, just feed to train_stdin data in format:

`(IN In)(DT the)(NN summer)(IN of)(CD 2005)(, ,)(DT a)(NN picture)(WDT that)(NNS people)(VBP have)(RB long)(VBN been)(VBG looking)(RB forward)(IN to)(-NONE- *T*-1)(VBD started)(-NONE- *-2)(VBG emerging)(IN with)(NN frequency)(IN in)(JJ various)(JJ major)(NNP Hong)(NNP Kong)(NNS media)(. .)`

`(IN With)(PRP$ their)(JJ unique)(NN charm)(, ,)(DT these)(RB well)(HYPH -)(VBN known)(NN cartoon)(NNS images)(RB once)(RB again)(VBD caused)(NNP Hong)(NNP Kong)(TO to)(VB be)(DT a)(NN focus)(IN of)(JJ worldwide)(NN attention)(. .)`

`(DT The)(NN world)(POS 's)(JJ fifth)(NNP Disney)(NN park)(MD will)(RB soon)(VB open)(IN to)(DT the)(NN public)(RB here)(. .)`

nlp4go's People

Contributors

korobool avatar demyan avatar fabregas avatar develer avatar gitter-badger avatar

Watchers

James Cloos avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.