gocode is using gigabytes and gigabytes of memory

[stamblerre]

gocode is using gigabytes and gigabytes of memory

[stamblerre]

I would expect gocode to be a little slower in modules, but I haven't heard any reports as drastic as this. Is it usable or is it so slow that you don't get completions? Is your project extremely large? To check that it's not related to Atom, do you mind testing it out on the command line? You can run gocode on the command line with gocode -in /path/to/file.go autocomplete /path/to/file.go <offset>.

[tekkamanendless]

The problem seems to appear in parallel contexts. I captured calls that Atom was making and ran them through gocode on the command line. Serially, there is no problem and everything works somewhat slowly, but as expected. In parallel (I ran 30 at a time for this test), gocode's memory usage ballooned to over 3GB. After completing all of the requests, it maintains that memory footprint.

My guess is that in the Atom use case, it's calling gocode more or less for every character that I type, and because the run time is longer due to the known caching deficiency in this fork, it runs into the memory-ballooning problem.

In some of my other Go projects, I know that I've had to manually call the garbage-collector after an individual task was completed because Go just wouldn't free up that memory for whatever reason. Maybe that's the case here.

Anyway, I can provide my exact code and the script to reproduce my memory issue. My project is just five files for a grand total of 45KB. I'll see if I can make a smaller reproducer for you.

[tekkamanendless]

Simple reproducer:

/tmp/reproducer/go.mod

module example.com/memory-issue

require github.com/davecgh/go-spew v1.1.1

/tmp/reproducer/main.go

package main

import (
        "fmt"

        "github.com/davecgh/go-spew/spew"
)

func main() {
        fmt.Printf("Hello, world!")
        spew.Dump("Some text")
}

type s1 struct {
        a string
}

Run this, and it uses up 0.5GB:

for (( i=0; i<10; i++ )); do $GOPATH/bin/gocode -in /tmp/reproducer/main.go autocomplete /tmp/reproducer/main.go 154 & done; wait;

Run this, and it doubles it to 1.0GB:

for (( i=0; i<20; i++ )); do $GOPATH/bin/gocode -in /tmp/reproducer/main.go autocomplete /tmp/reproducer/main.go 154 & done; wait;

Memory consumption seems proportional to the number of concurrent requests.

[stamblerre]

Thank you for the repro. This seems to be the issue mentioned in #22. I was hesitant to merge this change because of the additional dependency, but I will begin reviewing it.

[marcelino-m]

Same here, gocode is too slow in modules.

[stamblerre]

Just merged #22, so should be fixed.

[tekkamanendless]

Next up, I think that I'll need to patch go-plus for Atom to cancel requests faster (I don't think that it actually cancels requests after it sends the next ones).

[tekkamanendless]

Okay, short update: go-plus for Atom cancels its requests fairly quickly. I've patched gocode to call runtime.GC() after every request is handled, and while that helps, my gocode still balloons out to gigabytes and gigabytes (albeit much slower than before).

Is something still holding on to references to some of that stuff somewhere? I'll poke around with profiling it later.

[tekkamanendless]

Second update: calling debug.FreeOSMemory() instead of runtime.GC() after each request keeps gocode's memory from staying huge. It grows quickly while there are a bunch of requests in-flight, but it quickly drops back down to negligible amounts of memory immediately afterward.