Calls to plugin hang when passing large amounts of data back and forth

[knqyf263]

Calls to plugin hang when passing large amounts of data back and forth

[mathetake]

ack there's something memory leak might be happening in host generated code, as I confirmed the following test code can succeed fast with tinygo test -target wasi ./....

var p TestPlugin

func Test_In_TinyGo(t *testing.T) {
	ctx := context.Background()
	const entrySize = 10000
	// Initialize.
	response, err := p.PassData(ctx, host.PassDataRequest{
		Entries:  entrySize,
		Sequence: 0,
	})
	if err != nil {
		t.Fatal(err)
	}
	if len(response.Data) != entrySize {
		t.Fatalf("data size mismatch: %d != %d", len(response.Data), entrySize)
	}

	log.Printf("Size of data being passed %d (%d bytes)", len(response.Data), response.SizeVT())

	for i := uint64(0); i < 20; i++ {
		log.Printf("calling plugin %d", i)
		start := time.Now()
		response, err = p.PassData(context.Background(), host.PassDataRequest{
			Sequence: i,
			Entries:  entrySize,
			Data:     response.Data,
		})
		log.Printf("call execution time: %s", time.Since(start))
		if err != nil {
			t.Error(err)
			return
		}

		if len(response.Data) != int(entrySize) {
			t.Errorf("expected %d entries, got %d", entrySize, len(response.Data))
			return
		}
	}
}

[mathetake]

well, the test case is not enough to simulate the behavior -- I have to copy the entire PassDataRequest in each iteration. WIll fix it

[mathetake]

Updated, though it's till finishing in a decent time vs go test --- PASS: TestPlugin_MemoryMedium (80.04s):

$ tinygo test -target wasi ./...
--snip--
ok      github.com/cchamplin/plugin-bug/guest   9.114s

package main

import (
	"context"
	host "github.com/cchamplin/plugin-bug/plugin"
	"log"
	"strings"
	"testing"
	"time"
)

var p TestPlugin

func Test_In_TinyGo(t *testing.T) {
	ctx := context.Background()
	const entrySize = 10000
	// Initialize.
	response, err := p.PassData(ctx, host.PassDataRequest{
		Entries:  entrySize,
		Sequence: 0,
	})
	if err != nil {
		t.Fatal(err)
	}
	if len(response.Data) != entrySize {
		t.Fatalf("data size mismatch: %d != %d", len(response.Data), entrySize)
	}

	log.Printf("Size of data being passed %d (%d bytes)", len(response.Data), response.SizeVT())

	for i := uint64(0); i < 20; i++ {
		log.Printf("calling plugin %d", i)
		start := time.Now()
		req := &host.PassDataRequest{
			Sequence: i,
			Entries:  entrySize,
			Data:     make(map[string]*host.State, entrySize),
		}
		for k, v := range response.Data {
			state := &host.State{
				FieldA: strings.Clone(v.FieldA),
				FieldB: strings.Clone(v.FieldB),
				FieldC: strings.Clone(v.FieldC),
				FieldD: strings.Clone(v.FieldD),
				FieldE: strings.Clone(v.FieldE),
				FieldF: strings.Clone(v.FieldF),
				FieldG: strings.Clone(v.FieldG),
				FieldH: strings.Clone(v.FieldH),
				FieldI: strings.Clone(v.FieldI),
				FieldJ: strings.Clone(v.FieldJ),
				FieldK: strings.Clone(v.FieldK),
				FieldL: strings.Clone(v.FieldL),
				FieldM: strings.Clone(v.FieldM),
				FieldN: strings.Clone(v.FieldN),
			}
			req.Data[k] = state
		}

		response, err = p.PassData(context.Background(), *req)
		log.Printf("call execution time: %s", time.Since(start))
		if err != nil {
			t.Error(err)
			return
		}

		if len(response.Data) != int(entrySize) {
			t.Errorf("expected %d entries, got %d", entrySize, len(response.Data))
			return
		}
	}
}

[mathetake]

I am pretty confident that this is an TinyGo issue. Could you try https://github.com/wasilibs/nottinygc to see problems will be gone?

ref: tetratelabs/proxy-wasm-go-sdk#349 (comment)

[brennanjl]

Just stumbled across this, was wondering if anyone ever got this figured out @mathetake @cchamplin ?

Looking to use this library with Wazero in a data intensive application, with plugins written in TinyGo, but want to see if this got resolved.

[codefromthecrypt]

@brennanjl because large amounts of data have to be re-serialized into protobuf each time, it will be difficult to support with this model I think.

First, really consider what @mathetake said because I have seen one project drop 1/2 its overhead by using https://github.com/wasilibs/nottinygc

After that, if you still have issues, I think the choices are..

1. use this project, but make your own special host functions for the bulk data. (just the bulk is hard)
2. use a raw ABI approach instead of this project. (everything is hard)