bblfsh/sdk

Babelfish driver SDK

sdk

Babelfish SDK contains the tools and libraries required to create a Babelfish driver for a programming language.

Build

Dependencies

The Babelfish SDK has the following dependencies:

• Docker
• Go

Make sure that you've correctly set your GOROOT and GOPATH environment variables.

Install

Babelfish SDK gets installed using either Go:

$ go get -t -v github.com/bblfsh/sdk/v3/...

or make command:

$ make install

These commands will install bblfsh-sdk program at $GOPATH/bin/.

Contribute

The SDK provides scaffolding templates for creating a new language driver. These templates are converted to Go code that ends up in bblfsh-sdk tool. Use make bindata to update these templates:

$ make bindata
chmod -R go=r /home/user/bblfsh-sdk/etc/build; \
go run github.com/kevinburke/go-bindata/go-bindata/ \
        -pkg build \
        -modtime 1 \
        -nocompress \
        -prefix /home/user/bblfsh-sdk/etc/build \
        -o assets/build/bindata.go \
        /home/user/bblfsh-sdk/etc/build/...
go fmt ./assets/...
chmod -R go=r /home/user/bblfsh-sdk/etc/skeleton; \
go run github.com/kevinburke/go-bindata/go-bindata/ \
        -pkg skeleton \
        -modtime 1 \
        -nocompress \
        -prefix /home/user/bblfsh-sdk/etc/skeleton \
        -o assets/skeleton/bindata.go \
        /home/user/bblfsh-sdk/etc/skeleton/...
go fmt ./assets/...

You can validate this process has been properly done before submitting changes:

$ make validate-commit

If the code has not been properly generated, this command will show a diff of the changes that have not been processed and will end up with a message like:

generated bindata is out of sync
make: *** [Makefile:66: validate-commit] Error 2

Review the process if this happens.

On the other hand, If you need to regenerate proto and proteus files, you must run go generate from protocol/ directory:

$ cd protocol/
$ go generate

It regenerates all proto and proteus files under protocol/ and uast/ directories.

Usage

Babelfish SDK helps both setting up the initial structure of a new driver and keeping that structure up to date.

Creating the driver's initial structure

Let's say we're creating a driver for mylang. The first step is going to the location where we want the repository for the driver to be bootstrapped:

$ cd $GOPATH/src/github.com/bblfsh

Now the driver should be bootstrapped with bblfsh-sdk. This will create a git repository, and some directories and files required by every driver. They will be overwritten if they exist, like the README.md file in the example below.

$ bblfsh-sdk init mylang alpine
initializing driver "mylang", creating new manifest
creating file "manifest.toml"
creating file "Makefile"
creating file "driver/main.go"
creating file "driver/normalizer/normalizer.go"
creating file ".git/hooks/pre-commit"
creating file ".gitignore"
creating file ".travis.yml"
creating file "Dockerfile.build.tpl"
creating file "driver/normalizer/normalizer_test.go"
creating file "Dockerfile.tpl"
creating file "LICENSE"
managed file "README.md" has changed, overriding changes
$ git add -A
$ git commit -m 'initialize repository'

Note that this adds a pre-commit git hook, which will verify these files are up to date before every commit and will disallow commits if some of the managed files are changed. You can by-pass this with git commit --no-verify.

You can find the driver skeleton used here at etc/skeleton.

Keeping managed files updated

Whenever the managed files are updated, drivers need to update them. The update.go script can be used to perform some of those updates in managed files. For example, if the README template is updated, running go run update.go will overwrite it.

$ go run update.go
managed file "README.md" has changed, overriding changes

bblfsh-sdk doesn't update the SDK itself. Check Gopkg.toml for the target SDK version.

For further details of how to construct a language driver, take a look at Implementing the driver section in documentation.

Testing the driver

In order to run test for a particular dirver, change to it's directory and run:

$ go run ./test.go

This will:

• compile a "test binary" that parses content of the ./fixtures directory of the driver
• create a docker image with all dependencies, native driver and a test binary
• run this test binary inside a Docker container, using that image

The test binary first parses all source files in ./fixtures and generates a set of *.native AST files. Then the second set of test is run that uses *.native files, applies UAST annotations and normalizations to produce *.uast and *.sem.uast files.

If *.native files were already generated, the second stage can be run on the host without Docker container:

$ go test ./driver/...

Overall, SDK supports 4 different kind of tests for a driver:

• Native unit tests, parsing source files in ./fixtures and writing *.native. Runs only in Docker.
• UAST transformation unit tests, using *.native and writing *.uast files. Can be run both on host and in Docker.
• Integration tests, using content of ./fixtures/_integration*. Those are run in Docker using bblfshd.
• Benchmarks, using content of ./fixtures/bench_*.native. Can be run on the host or in Docker.

First two always run, benchmarks are only triggered by bblfsh-sdk test --bench or go test -bench=. ./driver/....

License

GPLv3, see LICENSE

sdk's Issues

Port annotation DSL predicates (BIP-5)

Port old annotation DSL predicates to a new transformation DSL.

Requires: #241

TODO:

• Self
• Children(HasInternalRole)
• Roles
• HasInternalType
• HasProperty
• HasToken
• And
• Not
• drop Children
• drop Descendants, DescendantsOrSelf
• drop HasInternalRole
• drop HasChild
• drop Or

Remove duplicated roles in the nodes

With the new agglutinative UAST it'll be hard to avoid some duplicated roles. They shouldn't have any effect, but they are un-classy and annoy some users.

It should be pretty easy to "uniq" the roles of a node in the SDK.

Improve driver bootstrapping

The initial code block described at babelfish documentation (creating a directory, a new git repo, etc.) could be done by bblfsh-sdk init command

Function arguments sorted in the wrong order

Doing the integration tests for the Python driver I noticed that the function arguments list, whose items has the internalRole "args" and Role MethodInvocationArguments (childrens of a MethodInvocation) are not sorting in the right order.

For this code:

print("something1", 42, somesymbbol)

I will attach the complete jsons for the Python AST and the generated UAST to this issue

unsorted_args_jsons.zip

but to see it clearly a (simplified) version of the Python AST part of the arguments is:

                  "args" : [
                     {
                        "LiteralValue" : "something1",
                        "ast_type" : "StringLiteral"
                     },
                     {
                        "ast_type" : "NumLiteral",
                        "NumType" : "int",
                        "LiteralValue" : 42,
                     },
                     {
                        "id" : "somesymbbol",
                        "ast_type" : "Name"
                     }
                  ],

While the (simplified) UAST generated is:

                     {
                        "Properties" : {
                           "NumType" : "0",
                           "internalRole" : "args"
                        },
                        "Roles" : [ 59, 54 ],
                        "Token" : "42",
                        "InternalType" : "NumLiteral"
                     },
                     {
                        "InternalType" : "Name",
                        "Token" : "somesymbbol",
                        "Roles" : [ 54, 0 ],
                        "Properties" : {
                           "internalRole" : "args",
                        },
                     },
                     {
                        "InternalType" : "StringLiteral",
                        "Properties" : {
                           "internalRole" : "args"
                        },
                        "Roles" : [ 58, 54 ],
                        "Token" : "something1",
                     }

As you can see, the arguments are not in the same order for the UAST. The Rule I'm using for them is:

		On(HasInternalType(pyast.Call)).Roles(MethodInvocation).Children(
			On(HasInternalRole("args")).Roles(MethodInvocationArgument),
			On(HasInternalRole("func")).Self(On(HasInternalRole("id"))).Roles(MethodInvocationName),
			On(HasInternalRole("func")).Self(On(HasInternalRole("attr"))).Roles(MethodInvocationName),
			On(HasInternalRole("func")).Self(On(HasInternalType(pyast.Attribute))).Children(
				On(HasInternalRole("id")).Roles(MethodInvocationObject),
			),
		),

The driver-included file `.sdk/etc/run.sh` uses bash instead of sh

This makes the driver Docker images need bash which they shouldn't need.

uast: merge roles for Assignment and AugmentedAssignment

Currently, we have:

	// Assignment represents a variable assignment or binding.
	// The variable that is being assigned to is annotated with the
	// AssignmentVariable role, while the value is annotated with
	// AssignmentValue.
	Assignment
	AssignmentVariable
	AssignmentValue

	// AugmentedAssignment is an augmented assignment usually combining the equal operator with
	// another one (e. g. +=, -=, *=, etc). It is expected that children contains an
	// AugmentedAssignmentOperator with a child or aditional role for the specific Bitwise or
	// Arithmetic operator used. The AugmentedAssignmentVariable and AugmentedAssignmentValue roles
	// have the same meaning than in Assignment.
	AugmentedAssignment
	AugmentedAssignmentOperator
	AugmentedAssignmentVariable
	AugmentedAssignmentValue

This feels quite redundant. I wonder if we can come up with a more succint way of annotating augmented assignments.

Refactor internal SDK types to use schema-less representation (BIP-4)

We will move current Node structure to protocol and will use maps and slices to represent AST nodes internally. Conversion to Node will happen in protocol package to preserve backward compatibility.

All the work happens in bblfsh/sdk@transform-dsl.

uast: define nonlocal access *usage* modifiers

These are modifiers that in general give an enclosed scope access to a variable in an outside scope. This could mean read/write or just read access.

This concept is similar to the existing "VisibleFrom" rules already on the AST but at the point of usage instead of declaration.

For example in Python we have "global" to allow an enclosed scope (usually a function definition) to modify a globally defined variable. There is also "nonlocal" for closures to be able to write variables of the enclosing scope; this concept (for closures) is usually called "captures" and it's also in C++11 and other languages supporting closures (in C++11 the capture is needed even for read access, in Python the read access is automatic and its used for obtaining write access).

We should investigate what other languages have for these access usage modifiers before defining how we'll do this (for example, are languages also supporting module/namespace/other access modifiers?

Odd issue with driver images fixed by reinstalling all docker images

Found while running the fixtures with a just build python-driver image and running the fixture regeneration: it didn't find the python 2 binary.

After deleting the docker image, I rebuild it and then when trying to install it into bblfshd it said:

"Installing python language driver from "bblfsh/python-driver:dev-21b075d-dirty"... Error, manifest unknown: manifest unknown".

I deleted everything (docker rmi (docker images -q) -f; and docker rm (docker ps -a -q) -f), redownloaded bblfshd's image and then it worked.

We should investigate this if it happens again.

generate up-to-date .proto files and gRPC service

We have to provide up-to-data proto files and gRPC service, which will be implemented by the bblfsh server.

Reconsider the expected precision of positioning

As @juanjux commented in slack:

We probably should relax the specifications of the positions to just provide what the native AST provides or we won't make a new driver per year. Future versions could then add optionally tokenizing information in the nodes.

Some children are randomly sorted on the pretty print

(I'll take a look at this, adding the issue to add a note on the sprint task for prefix/infix/postfix that depends on this being fixed).

How to reproduce:

• On a driver with integration tests (for example Python or Java), delete tests/.uast and tests/.native.
• make integration-test so these files are regenerated again.
• git commit adding the new files.
• Repeat the steps (delete, regenerate) without making any other changes.
• git diff

Example commit created just for debugging this.

support nested keys in ObjectToNoder

Take as an example the AST generated by babylon:

{
  "type": "SomeType",
  "loc": {
    "start": { "column": 1, "line": 1 },
    "end": { "column": 2, "line": 4 },
  },
  "offset": 1,
  "endOffset": 34
}

It'd be nice to be able to put as LineKey and so on loc.start.line instead of having to transform the AST.

Outdated drivers are treated as up-to-date on SDK updates

Currently drivers manifests has no mentions of version of SDK they were written for. It leads to a situation when driver might become outdated and incompatible, while still being listed as beta or even stable.

I propose to include new (required) field to manifest with semantic version of SDK. This field will be overwritten by bblfsh-sdk update and will be considered when interpreting driver status.

As always, minor version difference will not affect driver status, but major difference will automatically drop the status to inactive.

Implement the custom parser for Jupyter notebooks

This is to record the demand of data scientists to have *.ipynb parsed. There are quite a few already. I guess we should merge all the code cells and interpret them as a Python script.

@juanjux Do you think it should be added to Python driver? It is going to reuse 100% of the code.

annotations don't handle mutliple roles properly

When an annotation adds more than one role, only the first one is extracted in the annotation documentation. For example, the following annotations:

On(jdt.EnhancedForStatement).Roles(ForEach, Statement).Children(
	On(jdt.PropertyParameter).Roles(ForInit, ForUpdate),
	On(jdt.PropertyExpression).Roles(ForExpression),
	On(jdt.PropertyBody).Roles(ForBody),
),

generates the following annotation documentation:

| /self::\*\[@InternalType='CompilationUnit'\]//\*\[@InternalType='EnhancedForStatement'\] | ForEach |
| /self::\*\[@InternalType='CompilationUnit'\]//\*\[@InternalType='EnhancedForStatement'\]/\*\[@internalRole\]\[@internalRole='parameter'\] | ForInit |
| /self::\*\[@InternalType='CompilationUnit'\]//\*\[@InternalType='EnhancedForStatement'\]/\*\[@internalRole\]\[@internalRole='expression'\] | ForExpression |
| /self::\*\[@InternalType='CompilationUnit'\]//\*\[@InternalType='EnhancedForStatement'\]/\*\[@internalRole\]\[@internalRole='body'\] | ForBody |

while it should generate:

| /self::\*\[@InternalType='CompilationUnit'\]//\*\[@InternalType='EnhancedForStatement'\] | ForEach, Statement |
| /self::\*\[@InternalType='CompilationUnit'\]//\*\[@InternalType='EnhancedForStatement'\]/\*\[@internalRole\]\[@internalRole='parameter'\] | ForInit, ForUpdate |
| /self::\*\[@InternalType='CompilationUnit'\]//\*\[@InternalType='EnhancedForStatement'\]/\*\[@internalRole\]\[@internalRole='expression'\] | ForExpression |
| /self::\*\[@InternalType='CompilationUnit'\]//\*\[@InternalType='EnhancedForStatement'\]/\*\[@internalRole\]\[@internalRole='body'\] | ForBody |

Bootstraping a driver

Some things that I found missing (it could be maybe a lack of documentation)

Driver

Inside the driver, after running bblfsh-sdk prepare-build

• I think it should be autogenerated:
  • native directory if it does not exists,
  • .gitignore with a .sdk rule; if .gitignore exists without that rule, then append it.
• It should be great a hint in the root Makefile when running make if the SDK has not been yet installing, just saying something like:
  "It is needed to instal the bblfsh SDK before; it can be done with: bblfsh-sdk prepare-build"
• It should be explained somewhere that to start working with a driver, it is needed to run, in order:

bblfsh-sdk prepare-build
bblfsh-sdk init

SDK

Inside bblfsh/sdk project

• make test does not pass

> make test
--- FAIL: github.com/bblfsh/sdk/etc/skeleton/driver/normalizer :: TestNativeBinary
        Error Trace:    normalizer_test.go:18
	Error:		Expected nil, but got: &os.PathError{Op:"fork/exec", Path:"/opt/driver/src/build/native", Err:0x2}

• is it needed to run make all before doing anything? anyway tests keeps failing

It's possible to commit something, ignoring changes in managed files

related to #1 Can't commit after generating with bootstrap

steps to reproduce the bug:

• upgrade the RUNTIME_NATIVE_VERSION in the manifest.toml -> git add -A
• git commit -m "IT SHOULD FAIL" -> it fails, because a "managed file changed"
• bblfsh-sdk update -> it will generate a new README.md with the new manifest.toml specs
• git commit -m "IT SHOULD FAIL" -> it is COMMITTED!!! and it shouldn't, because you didn't added the README.md changes. (the new commit is broke, as you can see if you git co . and then bblfsh-sdk update --dry-run

possible solution:

The pre-commit must validate ONLY the staging area instead of the current working copy.
To do it, the process is:

1. stash everything not being in the staging area,
2. perform any validations
3. recover the stash done by (1)
4. exit with the exit code of the validations done by (2)

README: add development section

Add development section to README, explaining which files are generated, when to update them, etc.

The native version badge in the README.md mentions the driver name, instead of the native language

For instance, in case of the bash-driver, whose native driver is writen in Java, the batch should say java version: 1.8.... but instead it says bash version: 1.8.....

Add a rule to Makefile to install binaries

Currently, the way to get installed the sdk's binaries in $GOPATH/bin is running

go get -t -v ./...

Use an agglutinative language for Roles

This would make the Roles language more expressive and flexible, remove complexity, make it more extensive and better suited for language analysis.

There are various tasks that need to be done for this:

• Create a BIP describing the proposal: bblfsh/documentation#82
• Add versioning support for SDK
• Modify the SDK roles #171
• Update python driver to support new annotations
• Update java driver to support new annotations bblfsh/java-driver#47

ObjectToNoder doesn't map the nodes endpos to the EndPosition member

So they just show as properties.

https://github.com/bblfsh/sdk/blob/master/protocol/native/objecttonoder.go

Add the language to the ParseReponse

Will be useful when doing batch-parsing to have the filename in the response.

Can't commit after generating with bootstrap

Hook prevents the first time your want to commit README.md after generating it with bootstrap.

Driver images are not tagged as latest when a tag is released

Use Glide for the drivers to lock to specific SDK versions

Currently the drivers always link against ~master. We should add glide to the skeleton files and use it in the Go part of the driver build system.

sdk formats objects from AST weirdly

AST -> UAST

{"langSpecific": {"a": 1}} -> {"Properties": {"langSpecific": "map[a:1]"}}

Such format isn't very useful.

more info here: #213 (comment)

Make JSON decoder buffer size dynamic

A better solution for this kind of problems: #127

Roles numeric values do not match in Go and Proto

Roles numeric values do not match in Go and Proto. It seems they start at index 1 in Go and they start at index 0 in Proto.

It might be caused by a proteus issue: src-d/proteus#86
(or it might not)

Once we fix this, we need to build new versions of published drivers.

Update Go version to 1.9

This update will allow us to use type aliases, that can be used to map driver.Status to protocol.Status (and other similar cases).

make fails on a uninitialized git repository or repo with no commit

make fails on a uninitialized git repository or repo with no commit. It should initialize it or print a meaningful error.

➜  mylang-driver git:(master) ✗ make all
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Makefile:6: warning: overriding recipe for target 'test-native'
/home/smola/dev/demos/demo-2017-03-03/1_sdk/mylang-driver/.sdk/make/rules.mk:74: warning: ignoring old recipe for target 'test-native'
Makefile:10: warning: overriding recipe for target 'build-native'
/home/smola/dev/demos/demo-2017-03-03/1_sdk/mylang-driver/.sdk/make/rules.mk:85: warning: ignoring old recipe for target 'build-native'
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
+ mkdir -p /home/smola/dev/demos/demo-2017-03-03/1_sdk/mylang-driver/build
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
/bin/sh: 1: eval: cannot open /home/smola/dev/demos/demo-2017-03-03/1_sdk/mylang-driver/Dockerfile.build.tpl: No such file
/home/smola/dev/demos/demo-2017-03-03/1_sdk/mylang-driver/.sdk/make/rules.mk:67: recipe for target 'bblfsh/mylang-driver-build' failed
make: *** [bblfsh/mylang-driver-build] Error 2

Rule.Error should receive an error, not a string

That will make tests easier and errors much more informative.

Investigate if we can run `runc` without root

https://github.com/AkihiroSuda/containerd/blob/demo-rootless.20180116-1/docs/rootless.md

Implement transformation DSL (BIP-5)

We should implement shape-based transformation DSL as described in BIP-5.

All the work happens in bblfsh/sdk@transform-dsl.

TODO:

• Transformer
• Is
• Var
• Any
• And
• Not
• Check
• Object
• Out
• Arr
• OtherKeys
• Lookup
• Helpers

Return the language when using "autodetect" feature

When it is used "autodetect" feature, passing the filename and content in the request, it would be great to obtain the language as part of the response.

It could be something like:

message ParseResponse {
	option (gogoproto.goproto_getters) = false;
	option (gogoproto.goproto_stringer) = false;
	option (gogoproto.typedecl) = false;
	gopkg.in.bblfsh.sdk.v1.protocol.Status status = 1;
	repeated string errors = 2;
	google.protobuf.Duration elapsed = 3 [(gogoproto.nullable) = false, (gogoproto.stdduration) = true];
	gopkg.in.bblfsh.sdk.v1.uast.Node uast = 4 [(gogoproto.customname) = "UAST"];
	string language = 5;
}

instead of the current ParseResponse:

message ParseResponse {
	option (gogoproto.goproto_getters) = false;
	option (gogoproto.goproto_stringer) = false;
	option (gogoproto.typedecl) = false;
	gopkg.in.bblfsh.sdk.v1.protocol.Status status = 1;
	repeated string errors = 2;
	google.protobuf.Duration elapsed = 3 [(gogoproto.nullable) = false, (gogoproto.stdduration) = true];
	gopkg.in.bblfsh.sdk.v1.uast.Node uast = 4 [(gogoproto.customname) = "UAST"];
}

Expose the listing drivers endpoint for clients

As required by bblfsh/scala-client#68 and bblfsh/web#89, it would be needed to let the clients to fetch the list of installed drivers.

That API is part of the bblfshd/daemon/protocol, and it's currently only exported throwgh a unix socket for admin purposes so it can not be reached from the current clients.

Is there any plan to move the diver listing methods to the user network?

Here is the list of clients needing this:

• bblfsh/scala-client#68
• https://github.com/bblfsh/client-go/issues/53
• bblfsh/python-client#98

Add parent field into UAST Node in clients.

Sometimes it is important to keep it on hands.

If there is no such field you need to store all parents while performing some complicated UAST traversing.

Clarify error statuses returned by native drivers

Babelfish drivers may return 3 main statuses: OK, Error, Fatal.

Currently, there is no guarantee that Error responses will have a partial AST, since it's not tested in any of the drivers.

At the same time, Error status is not considered an error/exception in clients as reported in bblfsh/java-driver#77, thus clients might not be aware that they receive a partial AST.

We should either:
a) Make a special type of error/exception that user code should assert to accept a partial AST. In this case clients will receive an error in case of any syntax erros, which might be a desired behavior.
b) Mention current behavior in Babelfish docs, clarify that user should check the status.

Personally, I would prefer the first option.

Fill the Response.Language when the Request.Language was.

Currently, the server fills the Response.Language, either with the Request.Language if it was specified, or with the autodetected language if it wasn't. But when running against a driver container directly, the driver must at least do the first part. This should be implemented in the generic (golang) part of the drivers.

Add a tool to bblfsh-tools to regenerate integration test fixtures

Update drivers to use new DSL (BIP-5)

Update drivers to use new transformation DSL and new ToNode helpers.

Requires: #242 and #244

TODO:

• Go (bblfsh/go-driver#19)
• Java (bblfsh/java-driver#76)
• JavaScript (bblfsh/javascript-driver#19)
• PHP (bblfsh/php-driver#16)
• Python (bblfsh/python-driver#154)
• Ruby (bblfsh/ruby-driver#21)
• Bash (bblfsh/bash-driver#38)

uast: define async/await

Some languages (Jotlin, Python, Nim, etc) use async to qualify blocks (which are not always functions, sometimes they can be scoped blocks) that could run as interruptible coroutines. Usually those languages also have an await keyword for waiting for the completion of those blocks (similar to the join call/keyword for threads).

For functions this is usually done on the definition, with Go and Erlang instead having a similar keyword that is used on the call (we could have AsyncDefinition and AsyncCall for example).

We need to check several languages with built-in coroutine features and decide how to implement those roles in the AST.

Split ToNode into separate transformations (BIP-5)

Use new DSL to port old ObjectToNode transformations.

Requires: #241

TODO:

• InternalTypeKey
• OffsetKey, LineKey, ColumnKey
• EndOffsetKey, EndLineKey, EndColumnKey
• TokenKeys
• SpecificTokenKeys
• SyntheticTokens
• PromotedPropertyLists
• PromoteAllPropertyLists
• PromotedPropertyStrings
• TopLevelIsRootNode
• drop OnToNode
• drop Modifier
• drop IsNode

bblfsh-sdk pre-commit hook imcompatible with git < 2.7

The pre-commit hook installed by bblfsh-sdk has a compatibility issue with git < 2.7 because that's when the "--points-at" option to git branch was introduced (git/git@aa3bc55).

Is there a way to do something similar using older options?

More specific Parser.Tokenkeys or token-extracting annotations

Currently all the fields added to Parser.Tokenkeys are evaluated for token extraction for all the nodes. This has the problem that some nodes could have both of the fields, being only one of them the token field, but producing an error.

Two solutions could be used:

1. Change the data structure to optionally allow to specify an "internal_type" where the token extraction will be done for every specified token-field.

2. Add a new annotation that could extract a field as the token, e. g.:

// Node of internal type "Something" as the token in the "value" field:
On(HasInternalType(pyast.Something)).Roles(SomeRoles).TokenFrom("value")

Running container fails on GOPATH \w multiple dirs entries

Default GOPATH on GCE cloud console is "/home/alex/gopath:/google/gopath"

But running a build docker container for any driver fails on mounting such GOPATH

+ docker run --rm -t -u bblfsh:1000 -v /home/alex/java-driver:/opt/driver/src/ -v /home/alex/gopath:/google/gopath:/go -e ENVIRONMENT=bblfs
h/java-driver-build-with-go bblfsh/java-driver-build-with-go make test-driver-internal

docker: Error response from daemon: Invalid bind mount spec "/home/alex/gopath:/google/gopath:/go": invalid mode: /go.
See 'docker run --help'.

The SDK changes 0-columns and lines for 1 but checks empty files against 0 size

So if you pass an empty file, the TransformationParser will give this error:

column out of bounds: 1 [1, 0]

This error is produced both using a 0 as column index or an 1.

Build info on driver binaries is empty

$ docker run --rm bblfsh/python-driver:latest /opt/driver/bin/driver --help
Usage:
  /opt/driver/bin/driver [OPTIONS] <command>

Help Options:
  -h, --help  Show this help message

Available commands:
  parse-native
  parse-uast
  serve
  tokenize

Build information
  commit: 
  date:

Static compilation of sdk doesn't work

> make install
CGO_ENABLED=0 go get -t -v -ldflags '-extldflags "-static"' ./...
net
go install net: open /usr/lib/go/pkg/linux_amd64/net.a: permission denied
make: *** [Makefile:32: install] Error 1

This prevents drivers from building:

> make build
+ docker build -q -t bblfsh/bash-driver-build -f /home/abeaumont/go/src/github.com/bblfsh/bash-driver/.sdk/tmp/tmp.1510160514-853298827 .
+ docker run --rm -t -u bblfsh:1000 -v /home/abeaumont/go/src/github.com/bblfsh/bash-driver:/opt/driver/src/ -e ENVIRONMENT=bblfsh/bash-driver-build -e HOST_PLATFORM=Linux bblfsh/bash-driver-build make build-native-internal
+ docker build -q -t bblfsh/bash-driver-build-with-go -f /home/abeaumont/go/src/github.com/bblfsh/bash-driver/.sdk/tmp/tmp.1510160520-277170896 .
+ docker run --rm -t -u bblfsh:1000 -v /home/abeaumont/go/src/github.com/bblfsh/bash-driver:/opt/driver/src/ -v /home/abeaumont/go:/go -e ENVIRONMENT=bblfsh/bash-driver-build-with-go -e HOST_PLATFORM=Linux bblfsh/bash-driver-build-with-go make build-driver-internal
/bin/sh: /go/bin/bblfsh-sdk-tools: not found