twhoekstra/alpaca_kernel

Jupyter kernel to interact with a MicroPython over the serial REPL. Designed for use with the ALPACA board of the course NB2211 Electronic Instrumentation at the Delft Technical University.

ALPACA Kernel

Jupyter kernel to interact with a MicroPython over its serial REPL. Also with capabilities to do Ctrl-C and file transfer. Based on the Jupyter MicroPython Kernel.

Installation

First install Jupyter: http://jupyter.org/install.html (the Python3 version). It is strongly recommended to use the Anaconda Distribution

Via conda

conda install -c twh alpaca_kernel
python -m alpaca_kernel.install

Manual Installation (more maintainable)

Clone this repository to a directory using TortoiseGIT or with the shell command (ie on a command line):

git clone https://github.com/twhoekstra/alpaca_kernel

On Windows, Install this library (in editable mode) into Python3 using the shell command:

pip install alpaca_kernel

On Linux, you can install this library in editable mode using symlinks, which makes it easy to git pull a debugged version later:

pip install -e alpaca_kernel

Things can go wrong here, and you might need "pip3" or "sudo pip" if you have numerous different versions of python installed

Install the kernel into jupyter itself using the shell command:

python -m alpaca_kernel.install

This creates the small file ".local/share/jupyter/kernels/micropython/kernel.json" that jupyter uses to reference it's kernels

Post-Install

To find out where your kernelspecs are stored, you can type:

jupyter kernelspec list

Running

Now run Jupyter notebooks:

jupyter notebook

In the notebook click the New notebook button in the upper right, you should see your ALPACA kernel display name listed.

If you have a board already running MicroPython, plug it in, put the following command into the first cell and run it:

%serialconnect

If you are on Linux and don't have the correct permissions to access the Serial ports you will get a "permissions error". Fix it by adding yourself to the dialout and tty groups:

sudo usermod -a -G tty your_user_name
sudo usermod -a -G dialout your_user_name

If you are on Windows you need to have the right driver for the USB chip on the boad. Look for the USB connection in the Device List to identify the supplier and look for instructions relating to that supplier.

The notebook scans the serial ports (the COM-values in Windows) and tells you the possibilities and tries one of them. If it picks the wrong port you may need to be more specific and use the command:

%serialconnect --port=COM5

Further notes

There is a alpacademo.ipynb file in the directory you could look at with some of the features shown.

If a cell is taking too long to interrupt, it may respond to a "Kernel" -> "Interrupt" command.

Alternatively hit Escape and then 'i' twice.

To upload the contents of a cell to a file, write: %sendtofile yourfilename.py

as the first line of the cell

To do a soft reboot (when you need to clear out the modules and recover some memory) type: %reboot

Note: Restarting the kernel does not actually reboot the device.
Also, pressing the reset button will probably mess things up, because this interface relies on the ctrl-A non-echoing paste mode to do its stuff.

You can list all the functions with: %lsmagic

Debugging

For reference, the original developer notebooks here might be useful.

The system works by finding and connecting to a serial line and then issuing the enter paste mode command Ctrl-A (hex 0x01)

In this mode blocks of to-be-executed text are ended with a Ctrl-D (hex 0x04).

The response that comes back begins with an "OK" followed by the actual program response, followed by Ctrl-D, followed by any error messages, followed by a second Ctrl-D, followed by a '>'.

You can implement this interface (for debugging purposes) to find out how it's snarling up beginning with: "%serialconnect --raw" and then doing %writebytes -b "sometext" and %readbytes

Background

This had been proposed as an enhancement to webrepl with the idea of a jupyter-like interface to webrepl rather than their faithful emulation of a command line: micropython/webrepl#32

My first implementation operated a spawned-process asyncronous sub-kernel that handled the serial connection. Ascync technology requires the whole program to work this way, or none of it.
So my next iteration was going to do it using standard python threads to handle the blocking of the serial connections.

However, further review proved that this was unnecessarily complex if you consider the whole kernel itself to be operating asyncronously with the front end notebook UI. In particular, if the notebook can independently issue Ctrl-C KeyboardInterrupt signals into the kernel, there is no longer a need to worry about what happens when it hangs waiting for input from a serial connection.

Other known projects that have implemented a Jupyter MicroPython Kernel are:

• https://github.com/adafruit/alpaca_kernel
• https://github.com/willingc/circuitpython_kernel
• https://github.com/TDAbboud/mpkernel
• https://github.com/takluyver/ubit_kernel
• https://github.com/jneines/nodemcu_kernel
• https://github.com/zsquareplusc/mpy-repl-tool

In my defence, this is not an effect of not-invented-here syndrome; I did not discover most of these other projects until I had mostly written this one.

I do think that for robustness it is important to expose the full processes of making connections. For my purposes this is more robust and contains debugging (of the serial connections) capability through its %lsmagic functions.

Other known projects to have made Jupyter-like or secondary interfaces to Micropython:

• https://github.com/nickzoic/mpy-webpad
• https://github.com/BetaRavener/uPyLoader

The general approach of all of these is to make use of the Ctrl-A paste mode with its Ctrl-D end of message signals.
The problem with this mode is it was actually designed for automatic testing rather than supporting an interactive REPL (Read Execute Print Loop) system (citation required), so there can be reliability issues to do with accidentally escaping from this mode or not being able to detect the state of being in it.

For example, you can't safely do a Ctrl-B to leave the paste mode and then a Ctrl-A to re-enter paste mode cleanly, because a Ctrl-B in the non-paste mode will reboot the device.

alpaca_kernel's Issues

matplotlib.pyplot mathtype breaks kernel on MacOS

LaTeX formatting of label breaks plt.plot command

Minimal example

import matplotlib.pyplot as plt

plt.plot([1, 2, 3], [1, 2, 3], label='$f_{test}')

Current behavior

Prints the raw data. Cannot create plot. Especially annoying for large datasets.

{'label': '$f_{test}', 'fmt': '', 'data': None, 'scaley': True, 'scalex': True}[[0000803f0000004000004040], [0000803f0000004000004040]](3,)

Expected behavior

A plot.

Empty cells with just comments raise connection errors when no micro controller is connected

Overview

For Vocareum courses, answers in Jupyter Notebooks are typically entered in code cells formatted as:

# Your answer here 

###
### " Write your answer here "
###

Current behavior

When running these cells, and there is no micro controller connected, e.g. when a student is writing up the assignment after running the code, an error is thrown:

No serial connected
    %serialconnect to connect
    %esptool to flash the device
    %lsmagic to list commands

Expected behavior

The kernel recognized that the cell contains no code to run, so executing a cell does not throw any errors.