Giter Site home page Giter Site logo

naturalness / sensibility Goto Github PK

View Code? Open in Web Editor NEW
18.0 4.0 5.0 1.43 MB

Fixes Java syntax errors with LSTM neural networks! [proof-of-concept]

Home Page: https://eddieantonio.keybase.pub/santos2018syntax.pdf

License: Apache License 2.0

Makefile 0.72% Shell 1.70% Python 92.67% JavaScript 1.84% R 3.07%
ml4code lstm syntax-error syntax-checker java keras neural-network syntax hacktoberfest

sensibility's Introduction

Sensibility

https://travis-ci.org/naturalness/sensibility.svg?branch=master

Finds and fixes syntax errors. This is experimental software.

NOTE: Please use the saner2018 tag if you're looking to replicate the results from our SANER 2018 paper. Find the replication data on archive.org See Citation for how to cite this paper. Use the following to clone Sensibility with the correct branch:

git clone --branch=saner2018 https://github.com/naturalness/sensibility.git

Requirements

Sensibility requires:

  • Python 3.6.

For fixing Java files:

  • Java 8 SE

For fixing JavaScript files:

  • Node.JS >= 6.0 and ZeroMQ (macOS: brew install zeromq; Ubuntu: apt install libzmq-dev)

Researchers that wish to mine more repositories or languages will require:

  • A running Redis server.

Researchers wishing to evaluate the current methods will require:

Install

Activate a virtualenv, if that's your thing. Then,

pip install -e .

Usage

Once installed, there's one entry point to all the scripts and utilities included:

sensibility <SUBCOMMAND>

Most subcommands require the specification of a language, with the -l <LANGUAGE> option before the subcommand.

For example, to train Java models:

sensibility -l java train-list --help

Development

The following diagram visualizes the data flow. Subcommands to sensibility are in black rectangles; white ovals are products. Please contact the author to obtain the mistake database, which is only applicable when evaluating the Java models.

https://raw.githubusercontent.com/naturalness/sensibility/master/docs/dependencies.png

Tests

To run the tests, install tox using Pip, then run tox.

Mining repositories

  1. You must create a GitHub OAuth token and save it as .token in the repository root.

  2. Run redis-server on localhost on the default port.

  3. Use sensibility mine find-repos to get a list of the top ~10k repos:

    sensibility mine find-repos javascript | sort -u > javascript-repos.txt
    
  4. Use bin/enqueue-repo to enqueue repos to download:

    sensibility mine enqueue-repo < javascript-repos.txt
    
  5. Start one or more downloaders. These will dequeue a repo from the running Redis server and download sources:

    sensibility mine download
    

Evaluation

Type make experiments to train all of the models and evaluate each one. See libexec/experiments for more details.

Citation

If you use Sensibility in academic works, please use the following citation:

@inproceedings{santos2018,
    author={Santos, Eddie Antonio and
            Campbell, Joshua Charles and
            Patel, Dhvani and
            Hindle, Abram and
            Amaral, Jos{\'e} Nelson},
    booktitle={2018 {IEEE} 25th International Conference on Software Analysis, Evolution and Reengineering ({SANER})},
    title={Syntax and {Sensibility}: Using Language Models to Detect and Correct Syntax Errors},
    year={2018},
    month={Mar}}

License

Copyright 2016, 2017 Eddie Antonio Santos [email protected]

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

sensibility's People

Contributors

eddieantonio avatar luizperes avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

sensibility's Issues

Unable to open database file

Sensibility emits "(sqlite3.OperationalError) unable to open database file" when user chooses javascript after cloning project

(sensibility) Luizs-MacBook-Pro:sensibility luizperes$ sensibility mine download
INFO:ConcreteLanguageProxy:Language inferred from environment: javascript
Traceback (most recent call last):
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
    return fn()
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 328, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
    rec = pool._do_get()
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 1229, in _do_get
    return self._create_connection()
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
    return _ConnectionRecord(self)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
    self.__connect(first_connect_check=True)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 651, in __connect
    connection = pool._invoke_creator(self)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
    return dialect.connect(*cargs, **cparams)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 393, in connect
    return self.dbapi.connect(*cargs, **cparams)
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/luizperes/Projects/sensibility/bin/mine/download", line 35, in <module>
    downloader = Downloader()
  File "/Users/luizperes/Projects/sensibility/sensibility/miner/downloader.py", line 55, in __init__
    self.corpus = Corpus(writable=True)
  File "/Users/luizperes/Projects/sensibility/sensibility/miner/corpus.py", line 129, in __init__
    if self.empty:
  File "/Users/luizperes/Projects/sensibility/sensibility/miner/corpus.py", line 153, in empty
    metadata.reflect(self.engine)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 3791, in reflect
    with bind.connect() as conn:
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2091, in connect
    return self._connection_cls(self, **kwargs)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 90, in __init__
    if connection is not None else engine.raw_connection()
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2177, in raw_connection
    self.pool.unique_connection, _connection)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect
    e, dialect, self)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1465, in _handle_dbapi_exception_noconnection
    exc_info
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
    return fn()
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 328, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
    rec = pool._do_get()
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 1229, in _do_get
    return self._create_connection()
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
    return _ConnectionRecord(self)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
    self.__connect(first_connect_check=True)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/pool.py", line 651, in __connect
    connection = pool._invoke_creator(self)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
    return dialect.connect(*cargs, **cparams)
  File "/Users/luizperes/Projects/sensibility/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 393, in connect
    return self.dbapi.connect(*cargs, **cparams)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file

Field 'license' doesn't exist on type 'Repository'

GraphQL query is wrong according to https://developer.github.com/v4/object/repository/, because the field is licenseInfo rather than license.

(sensibility) Luizs-MacBook-Pro:sensibility luizperes$ sensibility mine download
INFO:ConcreteLanguageProxy:Language inferred from environment: javascript
INFO:download_worker:Downloader queue: q:work:010f6c39-7b0c-4616-b5ab-5c15bdee245c
INFO:download_worker:Fetching fbeline/Design-Patterns-JS
ERROR:download_worker:Error downloading "fbeline/Design-Patterns-JS
"
Traceback (most recent call last):
  File "/Users/luizperes/Projects/sensibility/sensibility/miner/downloader.py", line 79, in loop_forever
    self.do_job(job)
  File "/Users/luizperes/Projects/sensibility/sensibility/miner/downloader.py", line 105, in do_job
    repo = self.client.fetch_repository(repo_id)
  File "/Users/luizperes/Projects/sensibility/sensibility/miner/downloader.py", line 196, in fetch_repository
    """, owner=repo.owner, name=repo.name)
  File "/Users/luizperes/Projects/sensibility/sensibility/miner/downloader.py", line 230, in query
    raise ValueError(repr(response['errors']))
ValueError: [{'message': "Field 'license' doesn't exist on type 'Repository'", 'locations': [{'line': 15, 'column': 17}]}]
INFO:download_worker:Fetching fbrandel/ParisHilton.js
ERROR:download_worker:Error downloading "fbrandel/ParisHilton.js

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.