hovren/crisp

Camera-to-IMU calibration and synchronization toolbox

crisp's Issues

Translation between camera and IMU

Hi, @hovren, I have a few questions:
Can the crisp toolbox obtain the translation between camera and IMU?
How can I obtain the readout time of a camera?
Thank you in advance!

zncc: Division by zero

Hi, I have some troubles using crisp with my own samples. I took three sets of qvga@30fps video with 30Hz gyroscope and one set of 720p@60fps and 100Hz gyroscope and every one of those is giving me a Division by Zero error in the calibration.initialize(...) . I've tried to tune my setting values in your gopro_dataset_example.py, but the error is still there...

➜  examples python gopro_dataset_example.py dataset/samples/Set1/test.avi
Creating video stream from dataset/samples/Set1/test.avi
Creating gyro stream from dataset/samples/Set1/test_gyro.csv
Post processing gyroscope data to remove frequency spike noise
Creating calibrator
Estimating time offset and camera to gyroscope rotation. Guessing gyro rate = 30.00
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:67: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
Traceback (most recent call last):
  File "gopro_dataset_example.py", line 96, in <module>
    calibrator.initialize(gyro_rate=GYRO_RATE_GUESS)
  File "build/bdist.linux-x86_64/egg/crisp/calibration.py", line 128, in initialize
  File "build/bdist.linux-x86_64/egg/crisp/calibration.py", line 230, in find_initial_offset
  File "build/bdist.linux-x86_64/egg/crisp/timesync.py", line 107, in sync_camera_gyro
  File "build/bdist.linux-x86_64/egg/crisp/znccpyr.py", line 273, in find_shift_pyr
  File "build/bdist.linux-x86_64/egg/crisp/znccpyr.py", line 175, in zncc
ZeroDivisionError: division by zero

The debug output is saying :

DEBUG:crisp:Number of slices: 84
DEBUG:crisp:Generating optical flow magnitude. This can take minutes depending on video length
DEBUG:root:pyramid size = 7
DEBUG:root:size of first element 2243 
DEBUG:root:size of last element 35 

Do you have any idea of what I can be doing wrong?

Please add Cython to the list of dependencies.

understanding

I have several doubts, the first is how I calculate the matrix k,
the vector wc and omega,
readout is not 1/fps ¿why?
and finally where the stabilized video is exported (I can run the program perfectly but I do not see results)
I am using a gopro hero5.

stability of parameter estimation

Hi, I used the calibration toolbox to calibrate the data given by the paper, and the result was different every time.
In the experimental section of the paper, figure 4 shows the error relative to the set of reference parameters. The frequency error is within +/ -0.2, and the time offset error is between -0.008 and 0004. The results obtained within this range can achieve a nicely stabilised video, but the results of my calibration each time exceed the above range, resulting in the use of calibrated parameters can not generate a good stabilised video.

Here are some of the results I got(rotation.mp4):

Frequency(Hz) -1.245274 0.7984233 -1.291282
Offset(s) 0.004855215 0.004855141 0.004858229

Video Stabilization Method

Hi, @hovren
Thank you for your excellent work. After calibration, I need to stabilize the video. But I'm not sure which method to use. I want to stabilize the video on a UAV in real time. Do you have some recommendations?
Thank you in advance.

AttributeError: 'numpy.ndarray' object has no attribute 'sqrt'

Hi, when I run gopro_dataset_example.py with sample video in gopro-gyro-dataset， I got following errors in camera.py in line:

rn = np.sqrt((X[0,:] - wx)**2 + (X[1,:] - wy)**2)

AttributeError: 'numpy.ndarray' object has no attribute 'sqrt'

in function apply.

All three video: rotation.MP4, walk.MP4, rccar.MP4 will fall into this error after some successfully computation for previous calling.

I dumped the X data and found that X will be like below when reporting this error.
print(X) -->
[[array([-0.44746115])]
[array([0.02795225])]
[array([1.])]]

Could you help me found out why?
Thanks a lot!

Calibration fails: "Number of calls to function has reached maxfev = 800."

Trying to calibrate camera - gyroscope pair. Camera runs at 30 fps, Gyro at 100 Hz.

The example file 'gopro_dataset_example.py', when modified to suit our dataset, runs for 2 - 3 minutes and fails with the following message:

runfile('/crisp/gopro_dataset_example.py', args='/Set3/set3.mov', wdir='/crisp')
Creating video stream from /Set3/set3.mov
Creating gyro stream from /Set3/set3_gyro.csv
Post processing L3G4200D gyroscope data to remove frequency spike noise
Creating calibrator
Estimating time offset and camera to gyroscope rotation. Guessing gyro rate = 100.00
Running calibration. This can take a few minutes.
Calibration failed. Reason "Number of calls to function has reached maxfev = 800."
/crisp/gopro_dataset_example.py:101: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  print('Calibration failed. Reason "{}"'.format(e.message))
To exit: use 'exit', 'quit', or Ctrl-D.
An exception has occurred, use %tb to see the full traceback.

SystemExit: -2

Traceback:

Traceback (most recent call last):

  File "<ipython-input-12-64f97a132da6>", line 1, in <module>
    runfile('/crisp/gopro_dataset_example.py', args='/Set3/set3.mov', wdir='/crisp')

  File "/usr/local/lib/python2.7/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 714, in runfile
    execfile(filename, namespace)

  File "/usr/local/lib/python2.7/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 81, in execfile
    builtins.execfile(filename, *where)

  File "/crisp/gopro_dataset_example.py", line 102, in <module>
    sys.exit(-2)

SystemExit: -2

In the example python file, there are quite a few parameters which are not very obvious:

Care to explain these ones:

• CAMERA_DIST_CENTER
• CAMERA_DIST_PARAM
• CAMERA_READOUT

Also, in the *_reference.csv, please explain these column headers:

• Fg,
• offset
• rot_x
• rot_y
• rot_z
• gbias_x
• gbias_y
• gbias_z

What happens if I do not already know the gyro biases? Do we need to provide an estimate?

Error: 'flow_mag' referenced before assignment

Thanks for making this library public! I was testing it out with the following sample code:

import crisp

gyro_ms = 5.05050505048712

f = 1000.06549452
im_h, im_w = 1080, 1920
cx = im_w*0.5
cy = im_h*0.5

cam_mat = [[f, 0, cx],
           [0, f, cy],
           [0, 0, 1 ]]
disto = [-0.20100871814959385, 0.048642904244843899, 0, 0, -0.0063799856646590239]

gyro = crisp.GyroStream.from_csv("gyro.csv")
camera_model = crisp.OpenCVCameraModel((im_h, im_w), 59.94, 0, cam_mat, disto)
video = crisp.VideoStream.from_file(camera_model, "vid.mp4")

calibrator = crisp.AutoCalibrator(video, gyro)
calibrator.initialize(gyro_ms)
result = calibrator.calibrate() # Dict of calibrated parameters

print(result)

But I got the following error.

Traceback (most recent call last):
  File "calib.py", line 20, in <module>
    calibrator.initialize(gyro_ms)
  File "/home/matt/.local/lib/python3.8/site-packages/crisp/calibration.py", line 124, in initialize
    time_offset = self.find_initial_offset()
  File "/home/matt/.local/lib/python3.8/site-packages/crisp/calibration.py", line 226, in find_initial_offset
    time_offset = timesync.sync_camera_gyro(flow, frame_times, self.gyro.data.T, gyro_times, levels=pyramids)
  File "/home/matt/.local/lib/python3.8/site-packages/crisp/timesync.py", line 107, in sync_camera_gyro
    ishift = znccpyr.find_shift_pyr(flow_mag, gyro_mag, levels)
UnboundLocalError: local variable 'flow_mag' referenced before assignment

I am using python3.8 and installed crisp with python3 -m pip install crisp. I did have to change one function in as matplotlib doesn't have normpdf anymore. I changed it to from scipy.stats import norm instead.

Any idea how to fix this error?

AssertionError in calibration.py

I'm using the OpenCV camera model. After fixing several code issues and running crisp, I get the following error:

Creating video stream from /home/sergiu/Projects/test_sequences/building1.mp4
Creating gyro stream from /home/sergiu/Projects/test_sequences/building1_gyro.csv
Creating calibrator
Estimating time offset and camera to gyroscope rotation. Guessing gyro rate = 200.00
Traceback (most recent call last):
  File "parrot_ardrone_dataset.py", line 70, in <module>
    calibrator.initialize(gyro_rate=GYRO_RATE_GUESS)
  File "/home/sergiu/Projects/crisp/crisp/calibration.py", line 125, in initialize
    R = self.find_initial_rotation()
  File "/home/sergiu/Projects/crisp/crisp/calibration.py", line 290, in find_initial_rotation
    assert data.shape == (6, len(gyro_axes))
AssertionError

gyro_axes remains empty. I assume that estimating the initial rotation fails resulting in the error. Is this correct?

A bug in tracking.py

At line 208, I think "final_ok = np.intersect1d(ok_track, retracked_ok)" is wrong ,it should be "final_ok=ok_track[retracked_ok]". Because ok_track is forward_track and backward_track 's index, but retracked_ok is forward_first and backward_last 's index . They are not the same index. So they can't do the "intersect1d" operate. "retracked_ok" is "ok_track" 's index.

Fast Integrate Dependency

I was trying to install this, but it errors on build that it is missing a crisp/fastintegrate/fastintegrate.c file. I see that in the gitignore there as .c file is excluded. Where am I suppose to download this file so I can compile the project?

I am on ubuntu 14.04 and here is the error:

running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/crisp
copying crisp/cli.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/stream.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/rotations.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/calibration.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/znccpyr.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/videoslice.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/l3g4200d.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/remove_slp.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/tracking.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/__init__.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/timesync.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/imu.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/ransac.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/camera.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/pose.py -> build/lib.linux-x86_64-2.7/crisp
running build_ext
building 'crisp.fastintegrate' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/crisp
creating build/temp.linux-x86_64-2.7/crisp/fastintegrate
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c crisp/fastintegrate/fastintegrate.c -o build/temp.linux-x86_64-2.7/crisp/fastintegrate/fastintegrate.o
x86_64-linux-gnu-gcc: error: crisp/fastintegrate/fastintegrate.c: No such file or directory
x86_64-linux-gnu-gcc: fatal error: no input files
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 4

Edit: I am using the master branch.

Package with pip

It would be useful to be able to install using a simple pip install crisp.

Video Stabilization problem

Hi, I used the result of calibration toolbox to do video stabilization for the video in the paper. I found that according to the time offset in the calibration result, the stabilized video was very bad.
After scaling and offsetting the gyroscope data(as follow), I can get satisfactory results.

coeffies = [1.60190524e-03, -2.79751510e+00] # rotation.mp4
gyro_times = (gyro_times + coeffies[1]) / (1 - coeffies[0])

My question is, the video stabilization in the paper mentioned in the Readme directly use the time offset( 2.8333333333333335 as follow) of the calibration toolbox?

rotation.mp4 calibration result by the calibration toolbox:
gyro_rate = 853.660712254363
time_offset = 2.8333333333333335
gbias_x = -0.01001219994597313
gbias_y = -0.008216488048614853
gbias_z = 0.010610096301797737
rot_x = 0.022640457925138826
rot_y = -0.0068401509599219324
rot_z = -1.5426536394739456

Looking forward to your reply.

SVD did not converge error

I'm running crisp with my own camera and gyro data, it raises an error:

/usr/lib/python2.7/dist-packages/pkg_resources/init.py:1224: UserWarning: /home/lili/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).
warnings.warn(msg, UserWarning)
/usr/local/lib/python2.7/dist-packages/crisp-2.2-py2.7-linux-x86_64.egg/crisp/camera.py:181: RuntimeWarning: invalid value encountered in true_divide
Traceback (most recent call last):
File "gopro_dataset_example.py", line 75, in
calibrator.initialize(gyro_rate=GYRO_RATE_GUESS)
File "build/bdist.linux-x86_64/egg/crisp/calibration.py", line 127, in initialize
File "build/bdist.linux-x86_64/egg/crisp/calibration.py", line 256, in find_initial_rotation
File "build/bdist.linux-x86_64/egg/crisp/videoslice.py", line 52, in estimate_rotation
File "build/bdist.linux-x86_64/egg/crisp/rotations.py", line 306, in estimate_rotation_procrustes_ransac
File "build/bdist.linux-x86_64/egg/crisp/ransac.py", line 24, in RANSAC
File "build/bdist.linux-x86_64/egg/crisp/rotations.py", line 289, in
File "build/bdist.linux-x86_64/egg/crisp/rotations.py", line 72, in procrustes
File "/home/lili/.local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 1359, in svd
u, s, vt = gufunc(a, signature=signature, extobj=extobj)
File "/home/lili/.local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 99, in _raise_linalgerror_svd_nonconvergence
raise LinAlgError("SVD did not converge")
numpy.linalg.linalg.LinAlgError: SVD did not converge

How can I solve this?