src-d/ml-core

source{d} MLonCode foundation - core algorithms and models.

MLonCode Core Library

This library is the foundation for MLonCode research and development. It contains commonly used algorithms and tools to process the code-related data, such as Babelfish's UASTs, plain code text and other.

Contributions

...are welcome! See CONTRIBUTING.md and CODE_OF_CONDUCT.md.

License

Apache 2.0

Glossary

See here.

ml-core's Issues

Rename submit_event to record_event

As noted by @rporres, the name is misleading because we do not really send anything.

How to extract UAST's path contexts from code?

Given a python piece of code:

print("hello")

How can I extract its associated UAST contexts paths?, I tried to explore sourced library and see if there's a function for extracting the paths of a piece of code:

In:

from sourced.ml.core.extractors import bags_extractor
bags_extractor.Extractor?

Out:

Init signature: bags_extractor.Extractor(log_level=20)
Docstring:     
Converts a single UAST via `algorithm` to anything you need.
It is a wrapper to use in `Uast2Features` Transformer in a pipeline.
Init docstring:
Class constructor

:param log_level: logging level.
File:           ~/anaconda3/envs/sourced/lib/python3.6/site-packages/sourced_ml_core-0.0.3-py3.6.egg/sourced/ml/core/extractors/bags_extractor.py
Type:           type
Subclasses:     BagsExtractor, RoleIdsExtractor

and

In:

from sourced.ml.core.extractors import Extractor
Extractor.extract?

Out:

Signature: Extractor.extract(self, uast:bblfsh.node.Node)
Docstring: <no docstring>
File:      ~/anaconda3/envs/sourced/lib/python3.6/site-packages/sourced_ml_core-0.0.3-py3.6.egg/sourced/ml/core/extractors/bags_extractor.py
Type:      function

Also, I tried to:

from sourced.ml.core.utils import bblfsh
bblfsh.BblfshClient.parse(filename='/home/user/Downloads/script.py')

But I got:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-37-9dbc87e2a353> in <module>
      1 code = '''print("hi")'''
----> 2 bblfsh.BblfshClient.parse(filename='/home/user/Downloads/script.py')

TypeError: parse() missing 1 required positional argument: 'self'

After checking the available modules, I did not find a function for extracting the UAST path contexts. Is it possible to do it with sourced? Also I did not find any documentation (maybe someone can give me a pointer about where to find the docs?).

Empty docs

The built documentation is empty.

https://ml-core.readthedocs.io/en/latest/

Switch from BiLSTM to the modern attention architecture

Our current NN splitter is based on BiLSTM, which has problems with performance. We should leverage the recent advancements in deep learning and implement the new attention-based (seq2seq-like?) architecture of the model.

Stage 1 - research

Follow the paper, take the same dataset, and design the model. Calculate the metrics.

Stage 2 - production

Package the model, publish it on Modelforge.

Add Python package deployment to CI

We need to automatically deploy the project once we create a new tag.

Reference: https://github.com/src-d/style-analyzer/blob/master/.travis.yml#L69