nedbat/byterun

A Python implementation of a Python bytecode runner

Byterun

This is a pure-Python implementation of a Python bytecode execution virtual machine. I started it to get a better understanding of bytecodes so I could fix branch coverage bugs in coverage.py.

byterun's Issues

Introduction and README.rst

I volunteered to work on this project.
I think that the best way to understand Python is to
understand how the Python interpreter is written. Starting with cPython is just
too difficult. Byterun is a great place to start. A great way to understand software is to document it.
The first thing I would like to change
in the README.rst is to link to the excellent introductory article on Byterun.
http://www.aosabook.org/en/500L/a-python-interpreter-written-in-python.html

Does anyone have any preferences on how I write documentation?
In the comments? Separate docs? Read the docs? Elsewhere?
Is there anyone who would review my pull requests?

Of course one does not really understand until one has engaged with the software:
"I hear and I forget
I see and I remember
I do and I understand."

Then the next thing to do would be to process the pull requests.
I see there are three pull requests. In due course I plan to review them, although it will take a while until I get to it. This is not my day job. First I have to read and understand the code.
Is there anything else which needs to be done to Byterun?
Thanks for all the hard work already done on Byterun.
Chris

How to use these code? ModuleNotFoundError: No module named 'byterun'

CPython used for imported modules rather than byterun

Thanks for this great project -- I'm learning a lot from it. I'm using byterun as a test of a bigger project, and I found and fixed a bug:

oilshell/oil@62cd492

There are some irrelevant diffs there, but if you search for BUG FIX you will see this part. The core issue is that the MAKE_FUNCTION bytecode is not called if you do __import__. byterun uses the host __import__, which results in a native function, not a pyobj.Function. I also changed __repr__ of Function to make this more apparent.

Also of interest might be the speed tests at the end. I made two files: speed_main.py and speed.py. You will see that under byterun, using a function in library is much faster than using one in the same module, because it gets executed with CPython.

+ if isinstance(func, types.FunctionType):
+            defaults = func.func_defaults or ()
+            byterun_func = Function(
+                    func.func_name, func.func_code, func.func_globals,
+                    defaults, func.func_closure, self)
+        else:
+            byterun_func = func

If you are interested in a pull request, let me know. However I am highly confused about how you run byterun/__main__.py directly from the git repo with its relative imports? (and this is after 13 years of programming in Python, and being somewhat responsible for __main__.py beiing added in Python 2.5 or 2.6).

I guess you must either install it on the system or use some kind of wrapper? I think it is related to this:

http://stackoverflow.com/questions/11536764/how-to-fix-attempted-relative-import-in-non-package-even-with-init-py

I had to write a shell script wrapper than changed to the directory and used python -m byterun.__main__. Somehow changing PYTHONPATH didn't work.

Does not support list comprehension

It does not support list comprehension in Python 3.5. Other versions untested.

Trying to run """[x for x in [1,2,3]]"""：

<code object <listcomp> at 0x7fba81ddfed0, file "<python.py>", line 1>
  1           0 BUILD_LIST               0
              3 LOAD_FAST                0 (.0)
        >>    6 FOR_ITER                12 (to 21)
              9 STORE_FAST               1 (x)
             12 LOAD_FAST                1 (x)
             15 LIST_APPEND              2
             18 JUMP_ABSOLUTE            6
        >>   21 RETURN_VALUE

<code object <module> at 0x7fba80024660, file "<python.py>", line 1>
  1           0 LOAD_CONST               0 (<code object <listcomp> at 0x7fba81ddfed0, file "<python.py>", line 1>)
              3 LOAD_CONST               1 ('<listcomp>')
              6 MAKE_FUNCTION            0
              9 LOAD_CONST               2 (1)
             12 LOAD_CONST               3 (2)
             15 LOAD_CONST               4 (3)
             18 BUILD_LIST               3
             21 GET_ITER
             22 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             25 POP_TOP
             26 LOAD_CONST               5 (None)
             29 RETURN_VALUE
dissed, running
Caught exception during execution
Traceback (most recent call last):
  File "/usr/lib/python3.5/inspect.py", line 1088, in getfullargspec
    sigcls=Signature)
  File "/usr/lib/python3.5/inspect.py", line 2227, in _signature_from_callable
    return _signature_from_function(sigcls, obj)
  File "/usr/lib/python3.5/inspect.py", line 2103, in _signature_from_function
    kind=_POSITIONAL_OR_KEYWORD))
  File "/usr/lib/python3.5/inspect.py", line 2421, in __init__
    raise ValueError('{!r} is not a valid parameter name'.format(name))
ValueError: '.0' is not a valid parameter name

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/byterun/byterun/pyvm2.py", line 236, in dispatch
    why = bytecode_fn(*arguments)
  File "/usr/lib/python3.5/site-packages/byterun/byterun/pyvm2.py", line 930, in byte_CALL_FUNCTION
    return self.call_function(arg, [], {})
  File "/usr/lib/python3.5/site-packages/byterun/byterun/pyvm2.py", line 971, in call_function
    retval = func(*posargs, **namedargs)
  File "/usr/lib/python3.5/site-packages/byterun/byterun/pyobj.py", line 72, in __call__
    callargs = inspect.getcallargs(self._func, *args, **kwargs)
  File "/usr/lib/python3.5/inspect.py", line 1284, in getcallargs
    spec = getfullargspec(func)
  File "/usr/lib/python3.5/inspect.py", line 1094, in getfullargspec
    raise TypeError('unsupported callable') from ex
TypeError: unsupported callable
Traceback (most recent call last):
  File "/usr/lib/python3.5/inspect.py", line 1088, in getfullargspec
    sigcls=Signature)
  File "/usr/lib/python3.5/inspect.py", line 2227, in _signature_from_callable
    return _signature_from_function(sigcls, obj)
  File "/usr/lib/python3.5/inspect.py", line 2103, in _signature_from_function
    kind=_POSITIONAL_OR_KEYWORD))
  File "/usr/lib/python3.5/inspect.py", line 2421, in __init__
    raise ValueError('{!r} is not a valid parameter name'.format(name))
ValueError: '.0' is not a valid parameter name

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "python.py", line 173, in <module>
    """)
  File "python.py", line 166, in exec_python
    vm_value = vm.run_code(code)
  File "/usr/lib/python3.5/site-packages/byterun/byterun/pyvm2.py", line 145, in run_code
    val = self.run_frame(frame)
  File "/usr/lib/python3.5/site-packages/byterun/byterun/pyvm2.py", line 345, in run_frame
    six.reraise(*self.last_exception)
  File "/usr/lib/python3.5/site-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
TypeError: unsupported callable

Python3

Has anyone gotten this to run under Python 3? In Ubuntu, I tried 3.3 and 3.5 and 3.6... no luck, it always blew up at the same point. Actually, to be clear, Byterun sometimes worked to an extent, but the TESTS always blew up consistently, so that's what I mean by no luck. Has anyone gotten the tests to succeed?

Different workaround for http://bugs.python.org/issue19611

This is much less important than the last bug, but I had an issue with the workaround in the code. (This will probably never come up for anyone else, because I'm doing something unusual: using the old pure Python compiler module, not the CPython compile.c).

I think the way I am doing it is slightly simpler though. Since you are constructing
callargs = {'.0': args[0]} anyway, I think you can just check if there is one arg and it is called .0. The name is ignored for LOAD_FAST in favor of the index (lookup by number rather than name), but the name is still emitted.

oilshell/oil@65a504d#diff-702be8382fafa67a707b89a317abc246

- if PY2 and self.func_name in ["<setcomp>", "<dictcomp>", "<genexpr>"]:
+        # Different workaround for issue 19611 that works with
+        # compiler2-generated code.  Note that byterun does not use fastlocals,
+        # so the name matters.  With fastlocals, the co_varnames entry is just
+        # a comment; the index is used instead.
+        code = self.func_code
+        if code.co_argcount == 1 and code.co_varnames[0] == '.0':

Broken test of newest commit. Function object has no __code__ so not compatible with PY3.6

In the newest commit 62f9b1a, the test case will fail, because the test case use Function.code but the Function object provided by byterun not support it.

Idea: Using inside a debugger

Here's a crazy idea I had. Use this inside a debugger. (Of course, I prefer mine.) And this allows one to try running things in a second state for experimentation. To the extent that state is contained inside the 2nd interpreter (and doesn't touch external state), the whole thing can be rolled back and so on.

The specific place inside a debugger where I see great use is in working out the string argument in an eval or exec. you go to the parent frame and start executing instructions from some point just until the eval is called and then you can get an eval string. Yes, this borders on insanity.

It will be a while before I get a chance to try it, but I just thought I'd mention.

pyvm2.VirtualMachineError: unknown bytecode type: STORE_GLOBAL

Greetings. I'm new here, so please accept my apology in case I'm doing this wrong.

Did you not implement the opcodes for "global" variables on purpose, or is it an oversight?

Am I the only one ever using it?

Does not support Python 3.6

It throws IndexError: tuple index out of range.

Traceback (most recent call last):
  File "python.py", line 176, in <module>
    """)
  File "python.py", line 166, in exec_python
    vm_value = vm.run_code(code)
  File "/usr/lib/python3.6/site-packages/byterun/pyvm2.py", line 145, in run_code
    val = self.run_frame(frame)
  File "/usr/lib/python3.6/site-packages/byterun/pyvm2.py", line 318, in run_frame
    byteName, arguments, opoffset = self.parse_byte_and_args()
  File "/usr/lib/python3.6/site-packages/byterun/pyvm2.py", line 182, in parse_byte_and_args
    arg = f.f_code.co_consts[intArg]
IndexError: tuple index out of range

Actually, the intArg has become incredibly large, like 23040. I guess the format of the bytecode has changed.

closure Frame initialization uses wrong outer frame

The current version of byterun accesses cellvars through the frame the closure is executing under, instead of the frame it was defined in. The following code illustrates the problem:

def f():
x = ["foo"]
def inner():
x[0] = "bar"
return inner

def g(funcptr):
x = 5
def inner():
return x
y = funcptr()

g(f())

AssertionError: "'int' object does not support item assignment" != 'None'

__build_class__ is different in 3.4

Looks like we'll have some 3.3/3.4 compatibility issues if we decide to support that. __build_class__ is complaining that the constructor function it's passed is not the right kind of function.

CALL_FUNCTION_VAR and CALL_FUNCTION_VAR_KW in 3.5

There was a subtle change in order of argument for CALL_FUNCTION_VAR and CALL_FUNCTION_VAR_KW in CPython 3.5. In earlier versions *args is passed after keyword arguments. In 3.5 it is passed before them. See https://bugs.python.org/issue33216.

In addition you can pass multiple *args and **kwargs in 3.5. They are merged into a single tuple and dict with new opcodes BUILD_LIST_UNPACK and BUILD_MAP_UNPACK_WITH_CALL.

Questions about these lines of code

In function make_frame, I dont't understand these lines of code's effect.

    def make_frame(self, code, callargs={}, f_globals=None, f_locals=None):
        log.info("make_frame: code=%r, callargs=%s" % (code, repper(callargs)))
        # if f_globals is not None:
        #     f_globals = f_globals
        #     if f_locals is None:
        #         f_locals = f_globals
        # Can the above lines be modified like this
        if f_globals is not None and f_locals is None:
                f_locals = f_globals

        elif self.frames:
            f_globals = self.frame.f_globals
            f_locals = {}
        else:
            f_globals = f_locals = {
                '__builtins__': __builtins__,
                '__name__': '__main__',
                '__doc__': None,
                '__package__': None,
            }
        f_locals.update(callargs)
        frame = Frame(code, f_globals, f_locals, self.frame)
        return frame

has some problems when run code

intArg = byteint(arg[0]) + (byteint(arg[1]) << 8)
if byteCode in dis.hasconst:
arg = f.f_code.co_consts[intArg]

this is the code of function 'parse_byte_and_args', in this part, intArg is index out of range, i dont know why, can u help me, thanks

Blocks

The method manage_block_stack
https://github.com/nedbat/byterun/blob/master/byterun/pyvm2.py#L246-L307

mentions 4 different block types.

• loop
• finally
• setup-except
• with

and 5ldifferent whys

• yield
• continue
• break
• exception
• return

Maybe after closures this is the second hardest piece of code to understand. Of course there is lots of documentation on the web.

Generally Blocks are pretty obvious. Every time I indent, I create a block. I have to add them to the block stack. Understanding what happens with an exception is a bit harder to grok.

But this code really tangles them all together. There is not even a Block class, just a named tuple.

Block = collections.namedtuple("Block", "type, handler, level")

From a conceptual point of view, I think it would be much easier to create 4 different Block classes. Maybe they would have a shared base class. It would certainly be much easier to document.

I can understand why cPython did it this way. But my goal here is to understand the concepts, before I try to understand the optimizations.

So may I write the documentation as 4 different block classes? If I eventually write those classes, would you accept a pull request?

Warm Regards
Chris