robotpy/roborio-packages

Build files for RobotPy ipk packages

RobotPy RoboRIO Packages

This repository contains the build files used to build the RobotPy .ipk packages hosted at https://www.tortall.net/~robotpy/feeds/2022/. The current list of published packages can be found at that URL.

Installing a package (online)

NEW FOR 2020: You must import the RobotPy package signing key (fingerprint 46C8 9FEB 8973 8008 16F2 FA66 18C4 E7B9 5FBE B547). You can use the following command to install it:

curl https://www.tortall.net/~robotpy/feeds/robotpy.gpg | opkg-key add -

Create a .conf file in /etc/opkg (e.g. /etc/opkg/robotpy.conf) containing the following line:

src/gz robotpy https://www.tortall.net/~robotpy/feeds/2022

Here's a command you can copy/paste that does this:

echo "src/gz robotpy https://www.tortall.net/~robotpy/feeds/2022" > /etc/opkg/robotpy.conf

Then run opkg update. After you setup the opkg feed, you can run:

opkg install PACKAGENAME

Installing a package (offline)

You can use the RobotPy Installer Script to do offline opkg installs. First, download the package:

python3 installer.py download-opkg PACKAGENAME

Then, connect to the network with the RoboRIO, and install it:

python3 installer.py install-opkg PACKAGENAME

Building these packages yourself

Many of these packages are built directly on a roboRIO. Compiling them can eat up most of your RoboRIO's disk space, so you'll probably want to reimage it before using the RoboRIO in a competition.

Go into a directory and do this:

make ROBORIO=roborio-XXXX-frc.local all

Build Notes

• You will almost certainly want to setup passwordless login using an SSH key, as the compile process uses SSH to login to the roborio multiple times.

• Most of these packages can be compiled on a virtual machine, and the virtual machine won't run out of disk space or RAM quite so easily. See the roborio-vm repository for more details.

• Some packages use a lot of RAM, and your best bet is to use a swap device to allow it to complete. A USB memory stick works great for this.

  • On a linux host use cfdisk to partition your stick
  • Use mkswap to initialize the space.
  • Mount it on the roborio by using swapon

When adding new packages:

• Some packages have deeply recursive build functionality, if you have a weird segfault that occurs it might be because the process ran out of stack space (the default on the RoboRIO is 256k, whereas modern linux default to a few MB). In your build steps, you can set the stack size by executing ulimit -s 2048

roborio-packages's Issues

gfortran/ATLAS runtime error for 2016

Not totally unexpected, but I figured I'd try it with the existing libraries:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/site-packages/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/usr/local/lib/python3.5/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/usr/local/lib/python3.5/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/usr/local/lib/python3.5/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/usr/local/lib/python3.5/site-packages/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: /usr/local/lib/libatlas.so: undefined symbol: __aeabi_uidiv

There aren't instructions for updating ATLAS/libgfortran, otherwise I'd give it a try. It's probably not that bad though.. right? :p

Update ATLAS et al

I think the RoboRIO toolchain was upgraded, so this will need to be done.

Create an ARM VM to compile these in

It seems like this might not be that hard to do, and would allow us to build things like OpenCV and not have to mess around with the limited resources that the RoboRIO has on it.

Rename python packages

I propose that:

• python2 becomes python27
• python3 becomes python35

Let's take python35-numpy as an example. Because of the way our ipks work, numpy's files are always deployed to /usr/local/lib/python3.5, which means a 3.6 version of python3 wouldn't work correctly.

At the end of the day it doesn't really matter anyways, as we're not upgrading versions in the middle of the season anyways.

Build new packages for 2019

• Replace python 3.6 with 3.7
• I think there's a GCC upgrade, so we'll probably need an ATLAS rebuild...

Tar fails on macOS

Mac's tar does not support --owner or --group

Affects lines 15 & 16 in ipkg/Mk/Makefile:

tar -czvf control.tar.gz --owner=0 --group=0 control ${EXTRA_CONTROL}
cd data && tar -czvf ../data.tar.gz --owner=0 --group=0 . && cd ..

Workaround:

1. Install homebrew
2. Install gnu-tar
3. replace tar with gtar

Why is the python3 ipk so big?

Just curious, @PeterJohnson , the python3 ipk is something like 32MB? Given that the windows MSI files are ~15MB or so, and older RobotPy was ~10MB, that seems a bit big. Not that it matters too much (LabVIEW is a 4GB download, after all), but I am curious about it.

Adding Python 3.5

If I were to try and add python 3.5 here, what would that involve? Would it be better to add a seperate python35 package, or to upgrade the existing python3?

pip3 fails on roboRIO

This is very strange to me... any thoughts?

admin@roboRIO-1418:~# pip3 install -v ipython
Downloading/unpacking ipython
  Could not fetch URL https://pypi.python.org/simple/ipython/: connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)
  Will skip URL https://pypi.python.org/simple/ipython/ when looking for download links for ipython
  Could not fetch URL https://pypi.python.org/simple/: connection error: HTTPSConnectionPool(host='pypi.python.org', port=443): Max retries exceeded with url: /simple/ (Caused by <class 'http.client.CannotSendRequest'>: Request-sent)
  Will skip URL https://pypi.python.org/simple/ when looking for download links for ipython
  Cannot fetch index base URL https://pypi.python.org/simple/
  Could not fetch URL https://pypi.python.org/simple/ipython/: connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)
  Will skip URL https://pypi.python.org/simple/ipython/ when looking for download links for ipython
  Could not find any downloads that satisfy the requirement ipython
Cleaning up...
  Removing temporary dir /var/volatile/tmp/pip_build_admin...
No distributions at all found for ipython
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.4/site-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/local/lib/python3.4/site-packages/pip/req.py", line 1177, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/usr/local/lib/python3.4/site-packages/pip/index.py", line 277, in find_requirement
    raise DistributionNotFound('No distributions at all found for %s' % req)
pip.exceptions.DistributionNotFound: No distributions at all found for ipython

2020 RoboRIO uses xz compression for ipkg

.tar.gz still works, but there's definitely some space savings to be had if we switched.

Would need to update the ipkg.py tool and some of the makefiles.

Integrate compileall patch

Patch from python/cpython#4856 needs to be integrated into python package for 2018.

Then update pyfrc: robotpy/pyfrc#90

whl2ipk package versions not compatible with opkg

Needs some investigation, but see robotpy/robotpy-installer#12

Build ptvsd and pydevd with Cython extensions

See-also: robotpy/robotpy-wpilib#99

Remove mjpg-streamer in favor of using robotpy-cscore

Need to adjust the installer so it can install it.

Update Python for 2018

python3-dbg package doesn't work

Reading symbols from /usr/local/bin/python3...
warning: the debug information found in "/usr/local/bin/.debug/python3.4" does not match "/usr/local/bin/python3" (CRC mismatch).

Strip debug symbols from wheels?

Add this to whl2ipk?

os.getgroups() produces segmentation fault

Trying to execute the os.getgroups function in python produces a segmentation fault.

$ python3
>>> import os
>>> os.getgroups()

This works OK on my ubuntu desktop running python 3.4.1.

Originally filed by @dangyogi at robotpy/robotpy-wpilib#125

Bitbake error

I'm trying to make the cross-compiling of the library libgfortran on a virtual machine in a windows pc, to later build the library on a linux device. Following the guide I have encountered two problems:

• When I execute the command "git cherry-pick de2aa7a56790581406f219339c9022638cd47494", I get the error "fatal: bad object".

• When I execute "bitbake libgfortran", the following error appears:

ERROR: Function failed: do_compile (log file is located at /home/burruzola/nilrt/build/tmp_nilrt_3_0_x64-glibc/work/x86_64-linux/automake-native/1.14.1-r0/temp/log.do_compile.4677)
ERROR: Logfile of failure stored in: /home/burruzola/nilrt/build/tmp_nilrt_3_0_x64-glibc/work/x86_64-linux/automake-native/1.14.1-r0/temp/log.do_compile.4677
Log data follows:
| DEBUG: Executing shell function do_compile
| NOTE: make -j 6
| : && /bin/mkdir -p doc && { PATH='/home/burruzola/nilrt/build/tmp_nilrt_3_0_x64-glibc/work/x86_64-linux/automake-native/1.14.1-r0/build/t/wrap':$PATH && export PATH; } && /usr/bin/perl /home/burruzola/nilrt/build/tmp_nilrt_3_0_x64-glibc/work/x86_64-linux/automake-native/1.14.1-r0/automake-1.14.1/doc/help2man --output=doc/automake-1.14.1 automake-1.14
| help2man: can't get `--help' info from automake-1.14
| Try `--no-discard-stderr' if option outputs to stderr
| Makefile:3707: recipe for target 'doc/automake-1.14.1' failed
| make: *** [doc/automake-1.14.1] Error 255
| ERROR: oe_runmake failed
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_compile

Although a "build" folder is created with a subfolder for the machine inside it, the .ipk files are nowhere to be found as the bitbake command is not fully completed.

Get rid of 'Compiling...' messages from python package

Slows down the installation considerably.

Upgrade to Python 3.6

This will come out in mid December, and in particular there's a few performance improvements they've implemented. If it's like any other python upgrades, this shouldn't be a big deal.

Build a robotpy-gdb package for on-robot debugging?

The NI package depends on NI's python3, which might confuse users. Also, robotpy-installer doesn't seem to know how to install it.

Contemplate more automation for this repo

It's still largely a manual process, and there's not an easy way to stage stuff for testing without wiping out the production repo. One potential solution is using docker + my ODROID C2 as a build slave that auto builds on pushes to master (gitlab-ci actually makes this sort of thing easier).

gfortran doesn't seem to be distributed with 2016 opkg repos on RoboRIO

Not sure if there's someone at NI who could be contacted to fix this... I suspect not. This prevents usage of ATLAS, which is a dependency of numpy.

python3-dev error

admin@roboRIO-1418:~# opkg install python3-dev   
Installing python3-dev (3.4.2) to root...
Downloading http://www.tortall.net/~robotpy/feeds/2014/python3-dev_3.4.2_armv7a-vfp-neon.ipk.
Collected errors:
 * check_data_file_clashes: Package python3-dev wants to install file /usr/local/include/python3.4m/pyconfig.h
    But that file is already provided by package  * python3
 * opkg_install_cmd: Cannot install package python3-dev.

Add openssl support to python

https://peps.python.org/pep-0644/ requires OpenSSL > 1.1.1... NI only ships 1.0.2.

A different BLAS?

From @auscompgeek : https://github.com/flame/blis

Rewrite to not use make

The overall approach here is clunky. Make isn't really the right tool, as it's very difficult to do smart things like check for existence of data trees, remote package installs, etc. Maybe this would be better implemented as a (set of?) Python scripts?