go-python/gpython

gpython is a python interpreter written in go "batteries not included"

go-python

Naive go bindings towards the C-API of CPython-2.

this package provides a go package named "python" under which most of the PyXYZ functions and macros of the public C-API of CPython have been exposed.

theoretically, you should be able to just look at:

http://docs.python.org/c-api/index.html

and know what to type in your go program.

this package also provides an executable "go-python" which just loads "python" and then call python.Py_Main(os.Args). the rational being that under such an executable, go based extensions for C-Python would be easier to implement (as this usually means calling into go from C through some rather convoluted functions hops)

Install

With Go 1 and the go tool, cgo packages can't pass anymore additional CGO_CFLAGS from external programs (except pkg-config) to the "fake" #cgo preprocessor directive.

go-python now uses pkg-config to get the correct location of headers and libraries. Unfortunately, the naming convention for the pkg-config package is not standardised across distributions and OSes, so you may have to edit the cgoflags.go file accordingly.

 $ go get github.com/sbinet/go-python

If go get + pkg-config failed:

 $ cd go-python
 $ edit cgoflags.go
 $ make VERBOSE=1

Note: you'll need the proper header and python development environment. On Debian, you'll need to install the python-all-dev package

Documentation

Is available on godoc:

http://godoc.org/github.com/sbinet/go-python

Example:

package main

import "fmt"
import "github.com/sbinet/go-python"

func init() {
   err := python.Initialize()
   if err != nil {
          panic(err.Error())
   } 
}

func main() {
 	 gostr := "foo" 
	 pystr := python.PyString_FromString(gostr)
	 str := python.PyString_AsString(pystr)
	 fmt.Println("hello [", str, "]")
}

$ go run ./main.go
hello [ foo ]

TODO:

• fix handling of integers (I did a poor job at making sure everything was ok)

• add CPython unit-tests

• do not expose C.FILE pointer and replace it with os.File in "go-python" API

• provide an easy way to extend go-python with go based extensions

• think about the need (or not) to translate CPython exceptions into go panic/recover mechanism

• use SWIG to automatically wrap the whole CPython api ?

gpython's Issues

want to be a member of gpython

I read cpython twice. i think gpython is awesome, can i be a member of gpython? I will do my best.

Implement float is_integer method

python

>>> a = 10.0
>>> type(a)
<class 'float'>
>>> a.is_integer()
True

gpython

>>> a = 10.0
>>> type(a)
<class 'float'>
>>> a.is_integer()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
AttributeError: "'float' has no attribute 'is_integer'"

is_integer() method does not exist.

'#' in python shell make ... until input is received

Hyeockz:bin hyeockjinkim$ ./gpython 
Python 3.4.0 (none, unknown)
[Gpython dev]
- os/arch: darwin/amd64
- go version: go1.11.4
>>> #
... 
... 
... 
... 
... 
... 

In python shell, #, ... is printed until input is received.
Since # is single-line comment, ... should be printed only once in shell.

The output in python3 is:

Python 3.6.0 (v3.6.0:41df79263a11, Dec 22 2016, 17:23:13) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> #
... 
>>> 

Implement list pop method

Attribute errors occured using list.
​

Python (expected)

>>> a.pop()
4
>>>a
[1,2,3]
>>>a.pop(1)
2
>>>a
[1,3]

gpython (Actual)

>>> a = [1,2,3,4]
>>> a.pop()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
AttributeError: "'list' has no attribute 'pop'"

gpython: Can not retrieve a function attribute value

Expected

>>> def f():
...    f.x = 3
...    return f.x
...
>>> f()
3
>>>

Actual

>>> def f():
...    f.x = 3
...    return f.x
...
>>> f()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
  File "<stdin>", line 3, in f
    FIXME line of source goes here
SystemError: 'nil Dict in function'

gpython: Create benchmarks code

See #49

"print" should use __str__ or __repr__ when available

The builtin print() method is currently printing values like this (see https://github.com/go-python/gpython/blob/master/builtin/builtin.go#L192):

fmt.Printf("%v", v)

The result is that some python objects are printed differently than expect:

None: print {}
True: print true
False: print false

The print method should probably use str or repr when available. As an alternative all types implementing str should also implement String()

Implement slice object

Currently the slice object is not fully implemented.
Many features, including instance creation, are not fully implemented and must be implemented.

gpython/builtin/builtin.go

Line 91 in 8c361a8

// "slice": py.SliceType,

gpython/py/slice.go

Lines 16 to 18 in 8c361a8

	var SliceType = NewType("slice", `slice(stop) -> slice object
	"slice(stop)
	slice(start, stop[, step])

Expected result (cpython)

>>> slice(10)
slice(None, 10, None)
>>> slice(10).indices(0)
(0, 0, 1)

builtin: implement dir

right now, gpython fails with:

>>> import math
>>> dir(math)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
NameError: NameError: "name 'dir' is not defined"

The "get" function is not implemented in the Dict.

in gpython

>>>> d = {'a' : 1}
>>>> d.get('a')
Traceback (most recent call last):
  File "test.py", line 13, in <module>
    FIXME line of source goes here
AttributeError: "'dict' has no attribute 'get'"
2019/09/29 16:24:04 AttributeError: "'dict' has no attribute 'get'"

in python

>>>> d = {'a' : 1}
>>>> d.get('a')
1

Errors are suppressed in generator comprehensions

>>> list(i for x in range(10))
[]

Which should have output

>>> list(i for x in range(10))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <genexpr>
NameError: name 'i' is not defined

gpython should treat standard input as a file, not as input to the REPL

This definition works:

class X(object):
  def __init__(self, x):
    self.x = x
  def __str__(self):
    return "my name is %s" % self.x

This doesn't:

class X(object):
  def __init__(self, x):
    self.x = x

  def __str__(self):
    return "my name is %s" % self.x

Both are accepted by C Python.

runtime: Should support len(range(255)

During I am writing a proposal for boxed runtime type.
I found that gpython does not support it yet.

len(range(255))

print function's "end" option error in CPython

The following code writes different string to the stream than in CPython:

print(1,2,3, end="||\n")

Is there anything like GIL?

As we know, CPython implemention has a GIL.
How about gpython?

Implement bool.real, bool.imag bool.conjugate()

There are some attributes that should be implemented for gpython.

Expected

>>> a = True
>>> a.imag
0
>>> a.real
1
>>> a.conjugate()
1

Actual

>>> a = True
>>> a.image
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
AttributeError: "'bool' has no attribute 'image'"
>>> a.real
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
AttributeError: "'bool' has no attribute 'real'"
>>> a.conjugate()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
AttributeError: "'bool' has no attribute 'conjugate'"

Checking dict keys fails

In standard python

>>> x={"hello" : "world"}
>>> "hello" in x
True

In gpython

>>> x={"hello" : "world"}
>>> "hello" in x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
TypeError: "unsupported operand type(s) for iter: 'dict'"

`repr([1, 1.0])` is different from CPython

CPython:
repr([1,1.0]) # '[1, 1.0]'

GPython:
repr([1,1.0]) # '[1, 1]'

def a(x=3, b, c) should not be accepted

But accepted in gpython
it should be raised Syntax error.

cpython

>>> def a(x=3, b, c):
...     pass
...
  File "<stdin>", line 1
SyntaxError: non-default argument follows default argument

gpython

>>> def a(x=3, b, c):
...     pass
...
>>> a()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
TypeError: "a() missing 2 required positional arguments: 'x' and 'b'"

built-in objects __exit__ method is not picked up in contexts

I added the M__exit__ method to file to support "with open() as f" but the execution fails as following:

>>> with open("testfile") as f:
...   print("hello")
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
AttributeError: AttributeError: "'file' has no attribute '__exit__'"

The code in this case is only checking for the exit attribute instead of also checking for the implemented interface.

I see this comment:

 // exit := py.ObjectLookupSpecial(mgr, "__exit__")

But I am not sure of how ObjectLookSpecial could be implemented.

An easier way to do this, but it feels "wrong", would be to add an exit method in the object dictionary.

gpython: Implement iter()

TODO

• Implement callable_iterator
• Implement iter() builtin

callable_itereator should be implemented before implementing iter() builtin

Integrate ouroboros - Batteries Now Included

The ReadMe.md says:

`
It does not include very many python modules as many of the core modules are written in C not python. The converted modules are:

builtins
marshal
math
time
sys
`

What you need is the Ourobours library.

A standalone, pure Python implementation of the Python Standard Library. This allows Python default libraries to be used portably on any implementation of the Python virtual machine---not requiring CPython.

And here is the list of Modules.
https://github.com/beeware/ouroboros

Files 1488
Lines Of Code 419872

That should move this project very far forward.

generate python source code

given an ast, how does one generate the corresponding python source code

thank you

builtin: implement help

right now:

>>> import math
>>> help(math)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
NameError: NameError: "name 'help' is not defined"

meta: cross-check with CPython

as shown in #14, we should cross-check that our tests agree with some reference CPython version (presumably, CPython-3.)

so we should probably put some wiring so that each test.py script is run with both gpython and cpython.

Handle the return of non-integer value in __index__ function

An error should occur when returning a non-integer value from index.

Expected result (cpython)

>>> class C:
...     def __index__(self):
...         return 'a'
... 
>>> c = C()
>>> a = [1, 2 ,4]
>>> a[c]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __index__ returned non-int (type str)

Actual result (gpython)

>>> class C:
...     def __index__(self):
...         return 'a'
... 
>>> c = C()
>>> a = [1, 2, 4]
>>> a[c]
1

no such file or directory at first time gpython repl

I installed gpython with the command go get github.com/go-python/gpython and when I run in the first time in repl I get the next message error:
Failed to open history: open /home/matheus/.gpyhistory: no such file or directory
the repl do not stop working, and not presented no problem at first, but should be nice dont show this kind of error since that this does not hassles the gpython experience repl

Multiple Interpreters

Here is the PEP for multiple interpreters.
https://www.python.org/dev/peps/pep-0554/

It was rumored to be in Python3.7. Next I heard it would be in 3.8, then 3.9, now the Pep says provisional in 3.8, and they will make a decision about 3.9. I would not be surprised if multiple interpreters shipped first in gPython.

And I suspect that the cPython version will be one interpreter per OS thread. With gPython we could have thousands of interpreters.

I should also mention stackless.
https://github.com/stackless-dev/stackless/wiki

And finally the documents say that this is 1/5 th as fast as cPython. But I can easily get a server with 32 cores. And many applications now have little processing and lots of network I/O.

I just wonder what the killer app for this would be? It would be great to have a game, where a conference programming competition would be to win the multiplayer game, much like the snake competition at some recent Python conferences.

Warm Regards
Christopher Lozinski

NotImplemented comparison generate TypeError

TypeError occurs when comparing NotImplemented value from execution result.

Expected result (cpython)

>>> range(0).__ne__(2) == NotImplemented
True

Actual result

>>> range(0).__ne__(2) == NotImplemented
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
TypeError: "unsupported operand type(s) for ==: 'NotImplementedError' and 'NotImplementedError'"

gpython: leverage runtime/debug.ReadBuildInfo to infer version

starting with Go-1.12, there's a nifty function runtime/debug.ReadBuildInfo to extract build informations out of a Go binary.

we could leverage this to display:

$> gpython
Python 3.4.0 (none, unknown)
[Gpython (devel)]
- os/arch: linux/amd64
- go version: devel +4f13a9c5b1 Tue Oct 1 07:16:47 2019 +0000
>>> 

(notice the (devel) string that comes out of debug.ReadBuildInfo().Main.Version

this could be factored into a gpython/vm.Version() function:

// Copyright 2019 The go-python Authors.  All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.

// +build go1.12

package vm

import (
	"fmt"
	"runtime/debug"
)

// Version returns the version of Gpython and its checksum. The returned
// values are only valid in binaries built with module support.
//
// If a replace directive exists in the Gpython go.mod, the replace will
// be reported in the version in the following format:
//  "version=>[replace-path] [replace-version]"
// and the replace sum will be returned in place of the original sum.
//
// The exact version format returned by Version may change in future.
func Version() (version, sum string) {
	b, ok := debug.ReadBuildInfo()
	if !ok {
		return "", ""
	}

	const root = "github.com/go-python/gpython"

	modules := append([]*debug.Module{&b.Main}, b.Deps...)

	for _, m := range modules {
		if m.Path == root {
			if m.Replace != nil {
				switch {
				case m.Replace.Version != "" && m.Replace.Path != "":
					return fmt.Sprintf("%s=>%s %s", m.Version, m.Replace.Path, m.Replace.Version), m.Replace.Sum
				case m.Replace.Version != "":
					return fmt.Sprintf("%s=>%s", m.Version, m.Replace.Version), m.Replace.Sum
				case m.Replace.Path != "":
					return fmt.Sprintf("%s=>%s", m.Version, m.Replace.Path), m.Replace.Sum
				default:
					return m.Version + "*", m.Sum + "*"
				}
			}
			return m.Version, m.Sum
		}
	}

	return "", ""
}

Duplicated error type.

Got:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
TypeError: TypeError: 'all() takes exactly 1 argument (3 given)'

Expected:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
TypeError: 'all() takes exactly 1 argument (3 given)'

gpython: Optimize recursive call

Thanks to tengo and @ncw, we can compare the gpython with other interpreted which are implemented in Go.

https://github.com/d5/tengo#benchmark

fib(35) benchmark is slower than I expected, we can optimize this point later.
(tail call is faster than CPython, I can't believe it!)

all([]) should return false

From python documentation at https://docs.python.org/3/library/functions.html#all
Return True if all elements of the iterable are true (or if the iterable is empty).

[Question] How to implement object methods?

I want to try to implement hashlib (PR will follow).

For now I have implemented Type, module, module method and some "magic methods", but I cannot figure out how to implement object methods such as update, hexdigest etc.

Here is my WIP implementation.

Thanks in advance.

Found a problem with the args of the Dictionary "items" function.

Current Results
in gpython

>>>> d_list = {'a' : 1, 'b' : 2}
>>>> for k, v in d_list.items("hi"):
>>>>     print(k,':',v)
a : 1
b : 2

in python

>>>> d_list = {'a' : 1, 'b' : 2}
>>>> for k, v in d_list.items("hi"):
>>>>     print(k,':',v)
Traceback (most recent call last):
  File "test.py", line 2, in <module>
    for k, v in d_list.items("hi"):
TypeError: items() takes no arguments (1 given)

gpython/py/dict.go

Lines 30 to 39 in af17d7d

	func init() {
	StringDictType.Dict["items"] = MustNewMethod("items", func(self Object, args Tuple) (Object, error) {
	sMap := self.(StringDict)
	o := make([]Object, 0, len(sMap))
	for k, v := range sMap {
	o = append(o, Tuple{String(k), v})
	}
	return NewIterator(o), nil
	}, 0, "items() -> list of D's (key, value) pairs, as 2-tuples")
	}

I think we should add an error to this part of the code.

Implement range object

Expected Result

>>> range(10)
range(0, 10)
>>> range(10) == range(10)
True
>>> str(range(10))
'range(0, 10)'

Actual Result

>>> range(10)
<range instance at 0xc0002c8380>
>>> range(10) == range(10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
TypeError: "unsupported operand type(s) for ==: 'range' and 'range'"
>>> str(range(10))
'<range instance at 0xc0002c8c60>'

The __repr__, __str__, __eq__, and __ne__ functions of the range object are not implemented.

Ctrl+Z and Ctrl+C don't exit GPython on Windows 10

Currently it's hard to close the REPL on Windows using standard terminal.
On WSL Ubuntu it works with Ctrl+Z, in Windows nothing happens.

Recursive printing of REPR List, Dictionary, Tuple

Expected:

Gpython 3.4.0
>>> a = [1,2,3]
>>> a[0] = a
>>> a
[[...], 2, 3]

Actual behavior:

Gpython 3.4.0
>>> a = [1,2,3]
>>> a[0] = a
>>> a
runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow

The problem implementing print in gpython.

Even if sep is not required, the option must be inserted.

gpython

with open("testfile","w") as f:
    print("1234", end = "@@@", file = f)
with open("testfile","r") as f:
    print(f.read())`

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    FIXME line of source goes here
TypeError: 'print() argument 2 must be str, not file'
2019/09/26 11:21:21 TypeError: 'print() argument 2 must be str, not file'

when fixed code

with open("testfile","w") as f:
    print("1234", sep = '', end = "@@@", file = f)
with open("testfile","r") as f:
    print(f.read())

1234@@@

SyntaxError of global declaration don't stop program

a = 3
global a
print(a)

b = 2 + 5
print(b)

This code generate syntax error, but gpython doesn't stop program

Expected Result

Hyeockz:bin hyeockjinkim$ python3 g.py 
  File "g.py", line 2
    global a
SyntaxError: name 'a' is assigned to before global declaration

Actual Result

2019/09/09 00:08:36 name 'a' is assigned to before global declaration
3
7

SyntaxError in python3 is Error, not warning. So I think it should be modified.

Dict.__ne__ doesn't return NotImplemented

Dict.ne doesn't return NotImplemented.

gpython/py/dict.go

Lines 167 to 176 in af17d7d

	func (a StringDict) M__ne__(other Object) (Object, error) {
	res, err := a.M__eq__(other)
	if err != nil {
	return nil, err
	}
	if res == True {
	return False, nil
	}
	return True, nil
	}

If result of __eq__ is Not Implemented, __ne__ should return NotImplemented, not True

Expected result (python)

>>> a = {}
>>> a.__ne__(3)
NotImplemented

Actual result (gpython)

>>> a = {}
>>> a.__ne__(3)
True

How does format string work with `$`?

I'm currently implementing sorted, the signature is:

sorted(iterable, /, *, key=None, reverse=False)
    Return a new list containing all items from the iterable in ascending order.

    A custom key function can be supplied to customize the sort order, and the
    reverse flag can be set to request the result in descending order.

It already works as expected when passing either exactly one argument or all three.
When I only pass the first and the third, the third is interpreted as the second.

Python 3.4.0 (none, unknown)
[Gpython dev]
- os/arch: windows/amd64
- go version: go1.13
>>> l = [1.1, 1, 3]
>>> sorted(l)
[1, 1.1, 3]
>>> sorted(l, reverse=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
TypeError: "'bool' object is not callable"
>>> sorted(l, key=None, reverse=False)
[1, 1.1, 3]
>>> sorted(l, key=None, reverse=True)
[3, 1.1, 1]
>>>

I think my format string for py.ParseTupleAndKeywords should be "O|$Op:sorted", but this doesn't work too. Currently I have the following code:

func builtin_sorted(self py.Object, args py.Tuple, kwargs py.StringDict) (py.Object, error) {
	var iterable py.Object
	var keyFunc py.Object = py.None
	var reverse py.Object = py.False
	err := py.ParseTupleAndKeywords(args, kwargs, "O|Op:sorted", []string{"iterable", "key", "reverse"}, &iterable, &keyFunc, &reverse)
	if err != nil {
		return nil, err
	}
	l, err := py.SequenceList(iterable) // make a copy of the list
	if err != nil {
		return nil, err
	}
	err = SortInplace(l, keyFunc, reverse) // sort the copy in place
	if err != nil {
		return nil, err
	}
	return l, nil
}

Additionally Python doesn't allow sorted(list(), None, False), GPython allows this using the code above.

runtime: Wrong implementation with range

During I add a test for #40.
I found that this syntax does not work as I expected.

a = range(100, 0, -1)
len(a) <- 100
b = [a for e in a]
len(b) <- should be 100 but 0 in gpython

Implement set operation

These operations should be implemented

>>> a = {1, 2, 3}
>>> b = {2, 3, 4, 5}
>>> a | b
{1, 2, 3, 4, 5}
>>> a & b
{2, 3}
>>> a - b
{1}
>>> a ^ b
{1, 4, 5}
>>>

Implement set repr

Expected

>>> a = {1, 2, 3}
>>> a
{1, 2, 3}

Actual

>>> a = {1, 2, 3}
>>> a
<set instance at 0xc0000aa018>

Slice function of range type must be supported

a = range(10)
d = a[10:11]
print(d)

Unlike the list type, the range type does not support the slice function. slice in the range type must be supported.

Expected result

range(10, 10)

Actual result

TypeError: "unsupported operand type(s) for index: 'slice'"
2019/09/11 23:10:28 TypeError: "unsupported operand type(s) for index: 'slice'"

Cython support in go..

https://github.com/go-python/gopy

https://github.com/DataDog/go-python3

Might prove useful.

gpython: Display build information and support some builtin attribute

Likewise, PyPy3, if we can display build information such as commit information, go version.
it will be awesome.

Python 3.5.3 (fdd60ed87e941677e8ea11acf9f1819466521bf2, Apr 26 2018, 01:23:42)
[PyPy 6.0.0 with GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

Also, we need to implement copyright, creditsand license as a built-in attribute.

• copyright
• credits
• license
• build info

range: 'range' object is not subscriptable

Range object should support get_item operation :)

gpython

>>> a = range(10)
>>> a[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    FIXME line of source goes here
TypeError: "'range' object is not subscriptable"

python3

Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a = range(10)
>>> a[0]
0

py.Complex is missing __str__ and __repr__

Printing a complex number displays the py.Complex type instead of the real+imag values:

>>> print(3+1j)
<complex64 instance at %!p(py.Complex=(3+1i))>