symforce-org/symforce

Fast symbolic computation, code generation, and nonlinear optimization for robotics

symforce's Issues

Sparse matrices in Python

Generated python functions should support sparse matrix outputs (with scipy.sparse.csc_matrix). We should be able to generate factors with opt.Factor with sparse hessians, and linearize them into a wrapped linearized_sparse_factor_t

Docs cleanup

Make a pass over the docs to clean them up.

Components:

• Update everything to current best practices and APIs
• Add cross-references
• Make sure things are formatted nicely
• Make sure all doc comments in the code show up

Docstrings for pybind functions

Attempt to clean up the signature to Codegen.generate_function

Make sure we have docstrings in geo classes

Link back to symbolic functions

Rename safe replacements in sm to replace sm originals

Instead of adding a new function with a _safe suffix. This'll make it more readable, and make it harder to accidentally not notice the safe versions.

Some kind of check that epsilons are provided

Something that dynamically either lets you use a default epsilon of 0 (for playing around in notebooks/tutorials), or guarantee that it's either a symbol or explicitly set to 0 (for avoiding bugs in actual generated code)

Assume symbol is sm.symbol('epsilon')

Have custom Symbol subclass where evalf() == 0

Add heuristics for tangent_jacobians

For example, if a and b are geo.Rot3s, then tangent_jacobians(a * b, [b])[0] should be the identity matrix, but our current answer doesn't yield that.

Find ways to get expressions for jacobians with fewer ops than we currently use (using things like unit norm constraints and whatnot).

pybind functions in the docs

Fail on generating LCM types with eigen_lcm.Vector10d+ etc

This should fail at generation time, currently fails to find the expected bindings at compile time

We currently also don’t handle 2d matrices at all really. We generate all matrices as 1d storage vectors. eigen_lcm also only has square matrices and vectors, so if we want to generate matrices and not just storage, we’ll have to fail on non-square matrices.

Fast math functions codegen

Docs CD

Regenerate docs automatically from symforce master

Fix expected test failures

We've added a bunch of @unittest.expectedFailure's, we should make an effort to fix them

Add jacobian function to ops and non-Matrix geo and cam types

Reverse lookup of generated sym::Keys

Maybe constexpr function?

constexpr const char* KeyToSymbol(const sym::Key& key) {
  if (key == Var::VAR1) {
  } else {
    return "INVALID";
  }
}

Add function to sym::Values to take this and print with the symbol names

Codegen bug when generating python Values

Split SymforceGenCodegenTest to validate individual files or packages as they're generated

skymarshal doesn't work with pip install third_party/skymarshal

Only works with -e

Some level of support for std::complex

It would be nice if generating something using I and templating on std::complex didn't completely break, at least for scalar inputs and outputs

Update optimization tutorial notebook

cam in Python

Generate all the cam stuff in Python like it is in C++

Look at why shared types in python codegen function doesn't have prefixes

METIS CMake deprecation warning

CMake deprecation warnings from gklib and metis

CMake Deprecation Warning at build/_deps/gklib-src/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Deprecation Warning at build/_deps/metis-src/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

Symengine gives zero instead of NaN for singular substitution?

PyPI deploy

Deploy automatically to PyPI (or set up tooling to do this manually in a way that's documented and straightforward)

Get docs hosted

Get symforce docs set up hosted somewhere that's publicly visible

Feature parity between C++ and Python geo types

Fix float equality in symengine

In [1]: import sympy as sm

In [2]: sm.S(0) == 0
Out[2]: True

In [3]: sm.S(0) == 0.0
Out[3]: True

In [4]: import symengine as sm

In [5]: sm.S(0) == 0
Out[5]: True

In [6]: sm.S(0) == 0.0
Out[6]: False

Support python execution of C++ codegen

Is this the same as Cython-wrapped C++?

Create our own functions header used by all codegen

Here we can define approximate functions, pow behavior, and macros/configuration to turn on/off

Documentation on sym vs symforce namespaces

This has come up multiple times with pretty much everyone who's used symforce in python, we need to make this super clear

Create some mapping from ops to cycles

Cameras, especially PosedCamera, in C++ Values

Multiple people have requested this

Thoughts:

Store in Values

Don't want Retract for spherical, but want to store in Values

Nice to have generic "retract using data as a vector space"

VectorSpaceLieGroupOps to inherit from or something?

For image_size:

Make retract preserve image size

Probably doesn't work for from_tangent, but that should be fine (just set -1)

Can not implement compose, but custom implementation of retract to preserve not optimized stuff

Specify first n of storage vector which are optimized, rest are copied

Create geo.Matrix.unit_x() etc

pip install .

Run the entire build, including CMake, through setup.py, so that users can install from source with just pip install .. Useful reference might be the symenginepy build

Segfault in METIS

segfault in metis for exactly five optimized variables 🤯

best example I have currently, just dealing with two-vectors. the commented out jacobians/hessians look fine to me. any other chain length seems to work fine

import functools
import numpy as np

from symforce import geo
from symforce import sympy as sm
from symforce.opt.factor import Factor
from symforce.opt.optimizer import Optimizer
from symforce.values import Values

import sym


def between(x: geo.V2, y: geo.V2) -> geo.V2:
    return x - y


# if this is 4 or 6 or any other number I tried so far, it works
num_samples = 5
x_keys = [f"x{i}" for i in range(num_samples)]


values = Values()
for i in range(num_samples):
    values[x_keys[i]] = np.array([1, 2])
print(values)

factors = []

for i in range(num_samples - 1):
    factor = Factor(keys=[x_keys[i], x_keys[i + 1]], residual=between)
    factors.append(factor)

    # factor._generate_python_function(optimized_keys=factor.keys)
    # res, jac, hes, rhs = factor.generated_residual(values[x_keys[i]], values[x_keys[i + 1]])
    # print(jac)
    # import ipdb; ipdb.set_trace()

x_prior = geo.V2(1, 0)
prior_factor = Factor(
    keys=[x_keys[0]], name="prior", residual=functools.partial(between, y=x_prior)
)
factors.append(prior_factor)

# prior_factor._generate_python_function(prior_factor.keys)
# res, jac, hes, rhs = prior_factor.generated_residual(values[xs[0]])
# print(jac)

optimizer = Optimizer(factors=factors, optimized_keys=values.keys(), params=Optimizer.Params())

# segfault here in metis computing ordering
result = optimizer.optimize(values)

print(f"x_prior: {x_prior.T}")
print(result.initial_values)
print(result.optimized_values)
print(f"num iters: {len(result.iteration_stats)}")
print(f"final error: {result.error()}")

I added a print in SparseCholeskySolver<MatrixType, UpLo>::ComputePermutationMatrix, just before the failing line to ordering_(A_selfadjoint, inv_permutation_) to convert to dense and print:

Dense A_selfadjoint:
 1  0 -1  0  0  0  0  0  0  0
 0  1  0 -1  0  0  0  0  0  0
-1  0  1  0 -1  0  0  0  0  0
 0 -1  0  1  0 -1  0  0  0  0
 0  0 -1  0  1  0 -1  0  0  0
 0  0  0 -1  0  1  0 -1  0  0
 0  0  0  0 -1  0  1  0 -1  0
 0  0  0  0  0 -1  0  1  0 -1
 0  0  0  0  0  0 -1  0  1  0
 0  0  0  0  0  0  0 -1  0  1

with num_samples = 6, structure is the same

Dense A_selfadjoint:
 1  0 -1  0  0  0  0  0  0  0  0  0
 0  1  0 -1  0  0  0  0  0  0  0  0
-1  0  1  0 -1  0  0  0  0  0  0  0
 0 -1  0  1  0 -1  0  0  0  0  0  0
 0  0 -1  0  1  0 -1  0  0  0  0  0
 0  0  0 -1  0  1  0 -1  0  0  0  0
 0  0  0  0 -1  0  1  0 -1  0  0  0
 0  0  0  0  0 -1  0  1  0 -1  0  0
 0  0  0  0  0  0 -1  0  1  0 -1  0
 0  0  0  0  0  0  0 -1  0  1  0 -1
 0  0  0  0  0  0  0  0 -1  0  1  0
 0  0  0  0  0  0  0  0  0 -1  0  1

GDB backtrace shows the segfault in libmetis__genmmd

create_with_jacobians is broken with Values outputs

sym::Optimizer and other places still take 1e-9 epsilon

Generate C++ from Factor

First generate C++ from factor

Also be able to compile that factor from python and pass to python optimizer

std::bind-like option for Factor creation

std::bind(&sym::BetweenRot3<double>, /* a */ _1, /* b */ _2,
                  /* between_a_b */ geo::Rot3d::Identity(),
                  /* sigmas */ Eigen::Vector3d::Constant(between_sigma), epsilon, /* res */ _3,
                  /* jac */ _4),

sym::Bind(

Data buffers

Compare to ceres example

https://github.com/ceres-solver/ceres-solver/blob/master/examples/simple_bundle_adjuster.cc

Separate Manifold and LieGroup

We might want to have a Manifold spec for things that are differentiable manifolds that we might want to optimize over (e.g. camera calibrations) but that aren't Lie groups, and then have LieGroup be a Group and a Manifold

pip install symforce

Make it possible to pip install symforce from PyPI; this may not include making it easy to repeatedly do so, see #29

Generated `__init__.py` with multiple functions takes the last

A minimal example:

from symforce import codegen
from symforce import typing as T

def a(x: T.Scalar):
        return x + 1

def b(x: T.Scalar):
        return x + 2

output_dir = "<something>"
codegen.Codegen.function(a, config=codegen.PythonConfig()).generate_function(output_dir=output_dir)
codegen.Codegen.function(b, config=codegen.PythonConfig()).generate_function(output_dir=output_dir)

Then output_dir/python/symforce/sym/__init__.py will contain:

# -----------------------------------------------------------------------------
# This file was autogenerated by symforce from template:
#     python_templates/function/__init__.py.jinja
# Do NOT modify by hand.
# -----------------------------------------------------------------------------

from .b import b

Simple build integration for function codegen

Look into differences between SymPy and SymEngine CSE, and optimizations='basic'

Fix SymEngine Max derivative

This seems wrong:

sm.Max(x, y).diff(x).subs({x:y})

evaluates to 1/2

Also check in on derivatives of other things we added

ToStorage on Matrix does not match between C++ and Python

Eigen uses column-major and SymPy uses row-major, and in both cases we're just flattening/reshaping to get the storage, so the results are different. Should fix this

Faster interactive python workflow

Need to make this better, and then split out remaining items into additional TODOs

P1: The interactive workflow is too slow. The following 10 rotation chain takes 4.8 seconds to generate

2.9 seconds is rendering template, of which 2 seconds is calling black (lol), .8 seconds is calling string join inside of the template, and a small fraction is actually rendering

• We’re calling black.format_str 38 times. we should probably disable this for interactive use - flag in the language args of codegen?
  • I made an option to LanguageArgs to not autoformat. Perhaps controversial, but I’d propose to turn it off for the python Factor by default.
• Now str join is taking all the time of render template. What’s up with that?
• Why 38 times? We have 19 factors, so must be twice per factor.
  • What’s the second python file per factor? An __init__.py file from symforce/symforce/codegen/python_templates/function/__init__.py.jinja. Gross.
  • All prior and between factors should be the same expression. Can we only generate two?
• 2 seconds is in codegen_util.print_code. Seems like a lot of subs operations and list comprehensions, as well as some time in the sympy printer tree. Probably lots to optimize.

• A ton of time is being spent in subs in codegen_util.perform_cse, basically all initialization._get_subs_dict → _flatten_storage_type_subs → mostly python_util.scalar_like. We should work on reducing the calls to these or making them faster.