datadog/go-python3

Go bindings to the CPython-3 API

go-python3's Issues

can this work on windows ?

Describe what happened:
I try to use mingw64 to install go-python3 on windows, but failed:

go get -u -v github.com/DataDog/go-python3
github.com/DataDog/go-python3 (download)
github.com/DataDog/go-python3
# pkg-config --cflags  -- python3
Package python3 was not found in the pkg-config search path.
Perhaps you should add the directory containing `python3.pc'
to the PKG_CONFIG_PATH environment variable
No package 'python3' found
pkg-config: exit status 1

the PKG_CONFIG_PATH environment variable is already set to my python3.exe directory.

Describe what you expected:

Steps to reproduce the issue:

Goroutine calls Python but deadlocks

Describe what happened:

Does the project support Goroutine?
Recently, I encountered the following problems in the network programming project (simplified version):

• directory structure:

.
├── go.mod
├── go.sum
├── hello
│   ├── model
│   │   └── resnet18.pth
│   └── test.py
├── main.go
└── test

• target test.py file :

import torch
from torchvision import models


def Test(m):
    print("Call test function")
    model = models.resnet18(pretrained=True)
    if m == "save":
        torch.save(model, "./hello/model/resnet18.pth")
        return "model saved."
    else:
        return torch.load("./hello/model/resnet18.pth")

main.go:

import (
	"fmt"
	"github.com/datadog/go-python3"
	"os"
)

func init() {
	python3.Py_Initialize()
	if !python3.Py_IsInitialized() {
		fmt.Println("Error initializing the python interpreter")
		os.Exit(1)
	}
}

func main() {
	flagChan := make(chan bool , 1)
	testModule := ImportModule("./hello", "test")
	Test := testModule.GetAttrString("Test")
	args := python3.PyTuple_New(1)
	input := python3.PyUnicode_FromString("save")
	python3.PyTuple_SetItem(args, 0, input)
	res := Test.Call(args, python3.Py_None)
	repr, _ := pythonRepr(res)
	fmt.Println("[Main FUNC] res =", repr)
	go func() {
		fmt.Println("goroutine start...")
		args := python3.PyTuple_New(1)
		defer args.DecRef()
		input := python3.PyUnicode_FromString("load")
		python3.PyTuple_SetItem(args, 0, input)
		res := Test.Call(args, python3.Py_None)
		repr, _ := pythonRepr(res)
		fmt.Println("[Goroutine FUNC] res =", repr)
		fmt.Println("goroutine over")
		flagChan <- true
	}()
	<- flagChan
	args.DecRef()
	Test.DecRef()
	testModule.DecRef()
        python3.Py_Finalize()
	fmt.Println("main routine over")
	select {}
}

func ImportModule(dir, name string) *python3.PyObject {
	sysModule := python3.PyImport_ImportModule("sys") 	// import sys
	defer sysModule.DecRef()
	path := sysModule.GetAttrString("path")            // path = sys.path
	defer path.DecRef()
	dirObject := python3.PyUnicode_FromString(dir)
	defer dirObject.DecRef()
	python3.PyList_Insert(path, 0, dirObject) // path.insert(0, dir)
	return python3.PyImport_ImportModule(name)            // return __import__(name)
}

func pythonRepr(o *python3.PyObject) (string, error) {
	if o == nil {
		return "", fmt.Errorf("object is nil")
	}
	s := o.Repr()		
	if s == nil {
		python3.PyErr_Clear()
		return "", fmt.Errorf("failed to call Repr object method")
	}
	defer s.DecRef()
	return python3.PyUnicode_AsUTF8(s), nil
}

When I run it, sometimes it works perfectly, but sometimes it blocks python code:

Call test function
[Main FUNC] res = 'model saved.'
goroutine start...
Call test function
(It's always blocked here...)

Describe what you expected:

Call test function
[Main FUNC] res = 'model saved.'
goroutine start...
Call test function
[Goroutine FUNC] res = ResNet(....)
goroutine over
main routine over

I don't know if this is a problem with multiple goroutine resource preemption or not supporting multithreading, or maybe it's a problem with my go-python3 code.

Because I was writing a network communication project, I had to have multiple goroutine receive and send data (Model) enabled on the same node, and the entire python3 environment remained running until the node was shut down. I'm limited in what I can do, so I'd appreciate it if you could take a look

SyntaxError: invalid syntax (uuid.py, line 138)

Describe what happened:

python3.7 ubuntu18.04

run main.go get error
SyntaxError: invalid syntax (uuid.py, line 138)

demo.py

y = 6

import time
import uuid

def testF():
    global y
    y = y-2
    # print(uuid.uuid4())
    return 2*y*time.time()*uuid.uuid4()

if __name__ == "__main__":
    testF()

main.go

package main

import (
	"fmt"

	"github.com/DataDog/go-python3"
)

func main() {

	python3.Py_Initialize()

	defer python3.Py_Finalize()

	if !python3.Py_IsInitialized() {
		panic("Error initializing the python interpreter")
	}

	var err error


	err = python3.PySys_SetPath("/opt/gopwn/")
	if err != nil {
		panic("ERROR: Path set error.")
	}

	dir := "/usr/local/lib/python3.7/dist-packages"
	python3.PyRun_SimpleString("import sys\nsys.path.append(\"" + dir + "\")")

	pModule := python3.PyImport_ImportModule("test.demo.demo")
	if pModule == nil {
		panic("ERROR: Module not found.")
	}
}

Describe what you expected:

run main.go get error

`Traceback (most recent call last):
File "/opt/gopwn/test/demo/demo1.py", line 4, in
import uuid
SyntaxError: invalid syntax (uuid.py, line 138)
panic: ERROR: Module not found.

goroutine 1 [running]:
`
Steps to reproduce the issue:

Error / Stacktrace when python module fails

Describe what happened:
an error with the python script that cannot be caught - like a SyntaxError

Describe what you expected:
meaningful error reported in go

Steps to reproduce the issue:
run a python module with improper indenting

PyList_Insert and PyList_Append functions has an extra b

Describe what happened:

I want to add my site-packages to Path, but when I use PyList_Insert or PyList_Append function, I found an extra b before input :

func InsertBeforeSysPath(p string){
	sysModule := python3.PyImport_ImportModule("sys")
	path := sysModule.GetAttrString("path")
	pathRepr, err := pythonRepr(path)
	if err != nil {
		panic(err)
	}
	fmt.Printf("Path = %v\n", pathRepr)
	python3.PyList_Append(path, python3.PyBytes_FromString(p))
	pathRepr2, err := pythonRepr(path)
	if err != nil {
		panic(err)
	}
	fmt.Printf("Path2 = %v\n", pathRepr2)
}


// main
func main() {
	p := "/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python3.7/site-packages"
	InsertBeforeSysPath(p)
        ....
}

func pythonRepr(o *python3.PyObject) (string, error) {
	if o == nil {
		return "", fmt.Errorf("object is nil")
	}
	s := o.Repr()
	if s == nil {
		python3.PyErr_Clear()
		return "", fmt.Errorf("failed to call Repr object method")
	}
	defer s.DecRef()
	return python3.PyUnicode_AsUTF8(s), nil
}

the result is :

Path = ['/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python37.zip', '/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python3.7', '/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python3.7/lib-dynload', '/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python3.7/site-packages']
Path2 = ['/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python37.zip', '/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python3.7', '/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python3.7/lib-dynload', '/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python3.7/site-packages', b'/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python3.7/site-packages']

I tried Insert with the same result (b)

This will cause my next import module in main function to fail.

It works fine if not imported into Path (because it already exists), but I am afraid that porting to other machines will cause problems, so I hope you can give me some advice.

Describe what you expected:

Path2 = ['/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python37.zip', '/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python3.7', '/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python3.7/lib-dynload', '/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python3.7/site-packages', '/Users/xwj/opt/anaconda3/envs/fs_py37/lib/python3.7/site-packages']

Steps to reproduce the issue:

could not determine kind of name for C.PyDict_ClearFreeList

Describe what happened:
When using go get github.com/DataDog/go-python3 I get the response:

# github.com/DataDog/go-python3
../../go/pkg/mod/github.com/!data!dog/[email protected]/dict.go:141:13: could not determine kind of name for C.PyDict_ClearFreeList

But the package seems to work fine nevertheless.

Describe what you expected:
Probably no such error message?

Steps to reproduce the issue:
Simply using go get ...

Note: I use a M1 MacBook

How do I create an Object

Describe what happened:
I have this python object, and I want to store it as a variable. I do not know how to create a Py_Object that class in GoLang.

Describe what you expected:
object = ObjectName()
response = object.get_response()
print(response)

Steps to reproduce the issue:
I'm on a linux computer running ubuntu 20.04. This is more of a question than an issue.

fatal error: Python.h: No such file or directory #include "Python.h"

Describe what happened:

Python 3.7 Centos6.3
I can not use this
please help me

Describe what you expected:

Steps to reproduce the issue:

could not determine kind of name for C.PyImport_GetModule

Describe what happened:
i try to build code
stop here: vendor/github.com/DataDog/go-pyhon3/import.go:125:14 could not determine kind of name for C.PyImport.GetModule

Describe what you expected:

i install python3-devel.x86_64

Steps to reproduce the issue:

How to stop or terminate python execution?

Hi,

First, nice project 👍

How i can stop/terminate a python execution?

I want migrate my CI project (goci) from Go+Otto to Go+Python.

But i need some way to stop/terminate the execution.

Can you help me?

Thanks.

How do I create a Python bytes object

I wanted to pass a byte object to my python code but I have no idea with it:

my code:

        facedetect := ImportModule("./hello", "facedetect")
	getFaceStr := facedetect.GetAttrString("getFaceStr") // my python function to get face info

	webcam, _ := gocv.OpenVideoCapture(1)
	img := gocv.NewMat()
	webcam.Read(&img)
	buf := img.ToBytes() // the object to pass in python function
	face_buffer := python3.PyTuple_New(1)
	//buffer = buf
	python3.PyTuple_SetItem(face_buffer, 0,buf)  // how do I pass a Python bytes object
	face_res := getFaceStr.Call(face_buffer, python3.Py_None)
	log.Println(face_res)

thank you so much for your help !

Undefined reference to 'Py_True'

Describe what happened:
I got the following errors when I use 'go get github.com/DataDog/go-python3' to install this package on Ubuntu:
/tmp/go-build990175588/github.com/DataDog/go-python3/_obj/_cgo_main.o:(.data.rel+0x0):undefined reference to 'Py_True'
/tmp/go-build990175588/github.com/DataDog/go-python3/_obj/_cgo_main.o:(.data.rel+0x8):undefined reference to 'Py_None'
/tmp/go-build990175588/github.com/DataDog/go-python3/_obj/_cgo_main.o:(.data.rel+0x10):undefined reference to 'Py_False'

get PyObject's refcount

hi go-python3 community,

I have a quick question: Is there any easy way to get a PyObject's reference count when using this library? For example is there a way to access an object's ->ob_refcnt (preferably without doing any CGo lowlevel stuff)?

Any feedback would be much appreciated (also a "no" if not possible). Thanks a lot in advance.

How to wrap another Python after first install?

As far as I understand, when I build a binary that requires go-python3 then the resulting binary is linked dynamically to the Python that pkg-config found in the respective python3.pc file when installing go-python3 via go get. Is that right?

For example, I build a binary called gopython3, from your example program wrapping the entire interpreter. On MacOS otool displays the following for it:

$ otool -L gopython3
gopython3:
    /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/Python (compatibility version 3.7.0, current version 3.7.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)

Now, I have two questions:

a) Is it possible to build a standalone binary that contains the Python interpreter directly?

My guess is, that this is not possible. I am still quite new to Golang and also not really experienced with all this building and linking stuff since I come from Python :) So, I read this article and -as suggested in it- I tried to build my gopython3 with the following, which does not work:

$ CGO_ENABLED=0 go build .
go build github.com/DataDog/go-python3: build constraints exclude all Go files in /Users/user/go/pkg/mod/github.com/!data!dog/[email protected]

But I do not really understand why it does not work. Could it be made to work and I am just doing something wrong or is it really impossible?

b) How to point to another interpreter after installing go-python3 for the first time?

Currently, I just did as suggested in your documentation. I made sure that I have pkg-config installed and I checked that I have a corresponding configuration file around. Actually, it seems as if I have multiple of these since I have multiple versions of Python installed (Some that were installed by Homebrew as dependencies of some other software and some seem to come from Anaconda, which I mainly use for Python development.):

$ locate python3.pc
/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/pkgconfig/python3.pc
/usr/local/Cellar/python/3.7.7/lib/pkgconfig/python3.pc
/usr/local/Cellar/[email protected]/3.8.2/Frameworks/Python.framework/Versions/3.8/lib/pkgconfig/python3.pc
/usr/local/Cellar/[email protected]/3.8.2/lib/pkgconfig/python3.pc
/usr/local/anaconda3/lib/pkgconfig/python3.pc
/usr/local/anaconda3/pkgs/python-3.6.5-hc167b69_1/lib/pkgconfig/python3.pc
/usr/local/anaconda3/pkgs/python-3.7.0-hc167b69_0/lib/pkgconfig/python3.pc
/usr/local/anaconda3/pkgs/python-3.7.3-h359304d_0/lib/pkgconfig/python3.pc
/usr/local/lib/pkgconfig/python3.pc

I am unsure which of the python3.pc files is actually selected by pkg-config. The output from otool suggests that it is the the first one, isn't it?

Since I understood that I cannot build a standalone program (see above), which statically links Python, I thought that I would like at least to not require the end-users of my program to install Python but instead ship my own Python together with my Go built binary. I took this Python: indygreg/python-build-standalone, built it, and it contains a corresponding .pc file under python-build-standalone/python/install/lib/pkgconfig.

Now, I would like to link go-python3 to this Python instead to the one reported by otool above. But I do not know how to do that properly.

I tried deleting go-python3 and reinstall it as in the following:

$ PKG_CONFIG_PATH=python-build-standalone/python/install/lib/pkgconfig  go get github.com/DataDog/[email protected]
$ go build -a .

But my resulting gopython3 binary uses the same Python as before and not the one I intend to link. What am I doing wrong?

Thank you in advance for any hints!

Crashed when GOMAXPROCS > 1

I use the go-python3 to hook a snippet of python to the fasthttp package
(github.com/valyala/fasthttp)
It works like a charm if GOMAXPROCS==1 (with 10K requests/seconds)
But it crashed if GOMAXPROCS > 1 (such as 2, 4,...)
I have tried to acquire/release GIL, it crashes soon (not even 10 requests)
I have tried to use sync.Mutex.Lock/Unlock. It crashed too after 200 requests.
(Error: fatal error: unexpected signal during runtime execution)

Do you have any suggestion?

No module named xxx problem ,when the xxx is a so file ..

Describe what happened:
the first line of py code
from xxx import play

Describe what you expected:
the xxx is a so file.
Run the py script in command line is fine

Steps to reproduce the issue:
I read the the py script as string in go named code ,then python3.PyRun_SimpleString(code)
Not work , ModuleNotFoundError: no module named 'xxx'

Error: C.PyEval_ReInitThreads

Describe what happened:
github.com/DataDog/go-python3
../../go/pkg/mod/github.com/!data!dog/[email protected]/thread.go:53:2: could not determine kind of name for C.PyEval_ReInitThreads

Describe what you expected:

Steps to reproduce the issue:

error when go get and when run example.go

Hi I can run sbinet in my ubuntu18.0.4 with Python2. But now I want to use Python3. So I tried to go get this package. Then I got the following error:

I tried to run the example.go angway. Only to got the same error.
Can anyone help me?

Unable to call a function with parameters

I'm able to call a function in a module if does not have any parameters, like:

def func():
  print("OK")

but if the function has a non null parameter the function is not called and no error is given, like:

def func(val):
  print(val)

I'm using

myModule := python3.PyImport_ImportModule("mymodule")
f := myModule.GetAttrString("func")
if f == nil {
  panic("Error importing function")
}
f.Call(python3.PyList_New(1), nil)

python3.PyImport_ImportModule(name) will emit a fatal error when called the second time.

Describe what happened:
Environments:
On MacOS (Catalina Version 10.15.4)
Python3.7.6
Go1.13.8
I want to use go-python3 to invoke an algorithm written in Python3, but as described, an fatal error will generated when the second time I invoke this algorithm. From the output message, it seems that PyImport_ImportModule causes this error.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xa pc=0x91256a3]

runtime stack:
runtime.throw(0x4967a75, 0x2a)
        /usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:378 +0x47c

goroutine 41 [syscall]:
runtime.cgocall(0x4637740, 0xc000063c18, 0x48a4660)
        /usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000063be8 sp=0xc000063bb0 pc=0x4004d0b
github.com/DataDog/go-python3._Cfunc_PyImport_ImportModule(0x8061d90, 0x0)
        _cgo_gotypes.go:3780 +0x4a fp=0xc000063c18 sp=0xc000063be8 pc=0x462c2fa
github.com/DataDog/go-python3.PyImport_ImportModule(0x49501f5, 0x8, 0x0)
        /Users/zhao/go/pkg/mod/github.com/!data!dog/[email protected]/import.go:24 +0x87 fp=0xc000063c80 sp=0xc000063c18 pc=0x462e267
PPGServer/pkg/algo.ImportModule(0x4964926, 0x26, 0x49501f5, 0x8, 0x1)
        /Users/zhao/go/src/PPGServer/pkg/algo/ppg.go:42 +0x4cb fp=0xc000063d98 sp=0xc000063c80 pc=0x46332db
PPGServer/pkg/algo.CalcPre(0xc0003560c0, 0xd, 0x0, 0x0)
....

Here is the sample code.
A wrapper of PyImport_ImportModule:

// ImportModule will import python module from given directory
func ImportModule(dir, name string) *python3.PyObject {
	fmt.Println("python3.PyImport_ImportModule before")
	sysModule := python3.PyImport_ImportModule("sys") // import sys
	fmt.Println("python3.PyImport_ImportModule success")
	path := sysModule.GetAttrString("path")           // path = sys.path
	ob := python3.PyList_GetItem(path, 1)
	fmt.Println("check:", python3.PyUnicode_Check(ob))
	fmt.Println("path:", python3.PyUnicode_AsUTF8(ob))
	fmt.Println("sysModule.GetAttrString success")
	python3.PyList_Insert(path, 0, python3.PyUnicode_FromString("/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages"))
	python3.PyList_Insert(path, 0, python3.PyUnicode_FromString(dir))
	fmt.Println("After module insert:", python3.PyUnicode_AsUTF8(python3.PyList_GetItem(path, 0)))
	fmt.Println("module name:", name)
	return python3.PyImport_ImportModule(name) 
}

Each time the algorithm is called in a goroutine.

func CalcPre(dataFilePath string) (sbpI int, dbpI int) {
	python3.Py_Initialize()
	if !python3.Py_IsInitialized() {
		fmt.Println("Error initializing the python interpreter")
		os.Exit(1)
	}
	gstate = python3.PyGILState_Ensure()
	fmt.Println("Py_Initialize success")
	vbp := ImportModule("/Users/zhao/Desktop/lab/ppython", "value_bp")
	fmt.Println("ImportModule success")

	b := vbp.GetAttrString("estimate")
	fmt.Printf("[FUNC] b = %#v\n", b)
	bArgs := python3.PyTuple_New(1)
	python3.PyTuple_SetItem(bArgs, 0, python3.PyUnicode_FromString(dataFilePath))
	re := b.Call(bArgs, python3.Py_None)
	sbp := python3.PyTuple_GetItem(re, 0)
	dbp := python3.PyTuple_GetItem(re, 1)
	defer func() {
		python3.Py_Finalize()
		fmt.Println("python3.Py_Finalize()")
	}()
	sbpI = python3.PyLong_AsLong(sbp)
	dbpI = python3.PyLong_AsLong(dbp)
	python3.PyGILState_Release(gstate)
	return
}

func Calc(dataFilePath string) {
    CalcPre(dataFilePath)
}

Sample caller like this: go Calc("aaa.csv").

Describe what you expected:
We can call go Calc() many times without a fatal error.

Steps to reproduce the issue:
Use the code above and environments above, put these code into a goroutine, like go Calc("aaa.csv").For simplicity, you may just remove the algorithm part and just have the skeleton remained.

Hi, How to capture function return when use python3.PyRun_AnyFile

Describe what happened:
python file contians only one function return string, but I could't capture its return when use python3.PyRun_AnyFile

Describe what you expected:

Steps to reproduce the issue:

How might I pass in a http.ResponseWriter object as a *PyObject into PyObject_Call?

Describe what happened:
python3.PyTuple_SetItem(tuple, 0, w)
testfunc := pyMod.GetAttrString("serve")
pyret := testfunc.Call(tuple, ret)

running this results in:
cannot use w (type http.ResponseWriter) as type *python3.PyObject in argument to python3.PyTuple_SetItem

Describe what you expected:
I did not expect this to work, but would be interested in getting this to work so that http.ResponseWriter.Write could be called within the Python code. Is this possible?

Steps to reproduce the issue:

PyEval_ReInitThreads removed since python 3.8

Describe what happened:
PyEval_ReInitThreads() uses C.PyEval_ReInitThreads, but this symbol has been removed since python3.8.

Describe what you expected:
Please revise the code to check whether symbol C.PyEval_ReInitThreads exists. If not, just skip using this symbol in PyEval_ReInitThreads().

Steps to reproduce the issue:
With python 3.8, get go-python3 package.

Py_NewInterpreter

What do you think about adding support for Py_NewInterpreter and Py_EndInterpreter?
Would this be easier to use rather than using LockOSThread and PyGILState_Ensure?
I'm having issues with the latter, cannot make it work.

How could I Pass parameters from the Go side to the python file?

Describe what happened:
I have successfully run the listed examples on Ubuntu18.04.

Describe what you expected:
I wonder how could I pass parameters from the Go side to the python file, just like the way Go invoke the c file?

Steps to reproduce the issue:

how can i use tensorflow with library

i write this code,but it did not work
here is my code:
def update_model():
print("1.hello")
import tensorflow as ts
print("2.hello")

it can print frist hello,but can not print the second.

note:
i had use pip3 to install tensorflow library in my marchine, and this code can work in python3.7,but can not work when i use this lib to call this function.

could not determine kind of name for C.PyImport_GetModule

Describe what happened:
when the package installed with go install :
go install github.com/DataDog/go-python3
"gowork/src/github.com/DataDog/go-python3/import.go:125:14: could not determine kind of name for C.PyImport_GetModule"

why it happens?
Did i miss anything?

run on windows

Describe what happened:
how to run on windows

Describe what you expected:

I am waiting to run python codes in go project via windows 10.
Steps to reproduce the issue:

I can't compile on windows 10. What are package dependencies?

build error on debian

Describe what happened:
I got the following error when I compile the code

I've installed python3.7.6 and pkg-config --cflags -- python3 works fine

Describe what you expected:
No error

Steps to reproduce the issue: