strands-project/data_compression

Video encoding for 8 bit RGB images, 16 bit grayscale depth images and possibly more.

data_compression

Video encoding for OpenNI topics to record them to a rosbag and replay through reconstruction of point clouds.

Includes libav_image_transport by Dominique Hunziker: https://github.com/rapyuta/libav_image_transport.

Compression comparison

To put the compression gains of this library in context, here is a comparison to the default ROS image compression and the uncompressed images.

Both the ffv1 codec and the theora codec takes up quite a lot of CPU when compressing the OpenNI streams at 30 fps. The following CPU usages were measured on my Core i7 laptop.

data_compression's Issues

Adapt openni_launch nodes to work with strandsbag

We need to adapt the openni_launch nodes to work with the raw images coming from strandsbag. These have the same topics as the ones coming from the camera so it should be doable.

Associate depth and rgb frames

Associate depth and rgb frames and store them in the same place in the video parts.

Complete rgb support

This is basically done, but we have to decide if we want to associate the frames at this level, in that case we need one node as opposed to two separate, independent nodes.

No depth recorded

I am trying to record a bag file using

roslaunch rosbag_openni_compression record.launch file:=/path/to/file.bag camera:=head_xtion

It launches and outputs:

NODES
  /
    depth_compressor (image_transport/republish)
    recorder (rosbag/record)
    rgb_compressor (image_transport/republish)

ROS_MASTER_URI=http://bob:11311

core service [/rosout] found
process[depth_compressor-1]: started with pid [2566]
process[rgb_compressor-2]: started with pid [2571]
process[recorder-3]: started with pid [2576]
[ INFO] [1434642099.908496442]: Recording to /home/strands/test_rgbd_file.bag.
[ INFO] [1434642099.911241298]: Subscribing to /image_proc_head_xtion_rectify_mono/parameter_descriptions
[ INFO] [1434642099.920371009]: Subscribing to /rectify_depth_head_xtion/parameter_updates
[ INFO] [1434642099.927130821]: Subscribing to /image_proc_head_xtion_rectify_mono/parameter_updates
[ INFO] [1434642099.935951654]: Subscribing to /rectify_depth_head_xtion/parameter_descriptions
[ INFO] [1434642099.944202852]: Subscribing to /head_xtion/rgb/camera_info
[ INFO] [1434642099.952751981]: Subscribing to /image_proc_head_xtion_rectify_color/parameter_descriptions
[ INFO] [1434642099.961063612]: Subscribing to /rosout
[ INFO] [1434642099.969615944]: Subscribing to /image_proc_head_xtion_debayer/parameter_updates
[ INFO] [1434642099.979365094]: Subscribing to /rosout_agg
[ INFO] [1434642099.986902738]: Subscribing to /image_proc_head_xtion_debayer/parameter_descriptions
[ INFO] [1434642099.995034602]: Subscribing to /image_proc_head_xtion_rectify_color/parameter_updates
[ INFO] [1434642100.001341599]: Subscribing to /head_xtion/depth/camera_info
[ INFO] [1434642100.007597971]: Subscribing to /manager_head_xtion/bond
[ INFO] [1434642101.022842090]: Subscribing to /compressed_rgb/theora
[ INFO] [1434642101.035361454]: Subscribing to /compressed_rgb/theora/parameter_descriptions
[ INFO] [1434642101.042062744]: Subscribing to /compressed_rgb/theora/parameter_updates

but when I later play the bag, I can't get any pointclouds and get the message:

[ WARN] [1434641831.314939801, 1434641048.277004050]: [image_transport] Topics '/head_xtion/depth/image_raw' and '/head_xtion/depth/camera_info' do not appear to be synchronized. In the last 10s:
    Image messages received:      0
    CameraInfo messages received: 296
    Synchronized pairs:           0

rosbag info shows there are no compressed depth images in there:

strands@woody:~$ rosbag info test_rgbd_file.bag 
path:        test_rgbd_file.bag
version:     2.0
duration:    2:27s (147s)
start:       Jun 18 2015 16:23:56.71 (1434641036.71)
end:         Jun 18 2015 16:26:24.09 (1434641184.09)
size:        14.9 MB
messages:    14065
compression: none [20/20 chunks]
types:       bond/Status                           [eacc84bf5d65b6777d4c50f463dfb9c8]
             dynamic_reconfigure/Config            [958f16a05573709014982821e6822580]
             dynamic_reconfigure/ConfigDescription [757ce9d44ba8ddd801bb30bc456f946f]
             rosgraph_msgs/Log                     [acffd30cd6b6de30f120938c17c593fb]
             sensor_msgs/CameraInfo                [c9a58c1b0b154e0e6da7578cb991d214]
             theora_image_transport/Packet         [33ac4e14a7cff32e7e0d65f18bb410f3]
topics:      /compressed_rgb/theora                                        4366 msgs    : theora_image_transport/Packet        
             /compressed_rgb/theora/parameter_descriptions                    1 msg     : dynamic_reconfigure/ConfigDescription
             /compressed_rgb/theora/parameter_updates                         1 msg     : dynamic_reconfigure/Config           
             /head_xtion/depth/camera_info                                 4393 msgs    : sensor_msgs/CameraInfo               
             /head_xtion/rgb/camera_info                                   4392 msgs    : sensor_msgs/CameraInfo               
             /image_proc_head_xtion_debayer/parameter_descriptions            1 msg     : dynamic_reconfigure/ConfigDescription
             /image_proc_head_xtion_debayer/parameter_updates                 1 msg     : dynamic_reconfigure/Config           
             /image_proc_head_xtion_rectify_color/parameter_descriptions      1 msg     : dynamic_reconfigure/ConfigDescription
             /image_proc_head_xtion_rectify_color/parameter_updates           1 msg     : dynamic_reconfigure/Config           
             /image_proc_head_xtion_rectify_mono/parameter_descriptions       1 msg     : dynamic_reconfigure/ConfigDescription
             /image_proc_head_xtion_rectify_mono/parameter_updates            1 msg     : dynamic_reconfigure/Config           
             /manager_head_xtion/bond                                       883 msgs    : bond/Status                           (4 connections)
             /rectify_depth_head_xtion/parameter_descriptions                 1 msg     : dynamic_reconfigure/ConfigDescription
             /rectify_depth_head_xtion/parameter_updates                      1 msg     : dynamic_reconfigure/Config           
             /rosout                                                         18 msgs    : rosgraph_msgs/Log                     (9 connections)
             /rosout_agg                                                      3 msgs    : rosgraph_msgs/Log

The camera certainly publishes depth data:

strands@bobl:~$ rostopic hz /head_xtion/depth/image_raw
subscribed to [/head_xtion/depth/image_raw]
average rate: 29.933
        min: 0.031s max: 0.037s std dev: 0.00116s window: 29

but during record there is no /compressed_depth topic, but many depth compressed things:

strands@bobl:~$ rostopic list | grep compressed
/compressed_rgb/theora
/compressed_rgb/theora/parameter_descriptions
/compressed_rgb/theora/parameter_updates
/head_xtion/depth/image_rect/compressed
/head_xtion/depth/image_rect/compressed/parameter_descriptions
/head_xtion/depth/image_rect/compressed/parameter_updates
/head_xtion/depth/image_rect/compressedDepth
/head_xtion/depth/image_rect/compressedDepth/parameter_descriptions
/head_xtion/depth/image_rect/compressedDepth/parameter_updates
/head_xtion/depth/image_rect_meters/compressed
/head_xtion/depth/image_rect_meters/compressed/parameter_descriptions
/head_xtion/depth/image_rect_meters/compressed/parameter_updates
/head_xtion/depth/image_rect_meters/compressedDepth
/head_xtion/depth/image_rect_meters/compressedDepth/parameter_descriptions
/head_xtion/depth/image_rect_meters/compressedDepth/parameter_updates
/head_xtion/depth_registered/image_rect/compressed
/head_xtion/depth_registered/image_rect/compressed/parameter_descriptions
/head_xtion/depth_registered/image_rect/compressed/parameter_updates
/head_xtion/depth_registered/image_rect/compressedDepth
/head_xtion/depth_registered/image_rect/compressedDepth/parameter_descriptions
/head_xtion/depth_registered/image_rect/compressedDepth/parameter_updates
/head_xtion/rgb/image_color/compressed
/head_xtion/rgb/image_color/compressed/parameter_descriptions
/head_xtion/rgb/image_color/compressed/parameter_updates
/head_xtion/rgb/image_color/compressedDepth
/head_xtion/rgb/image_color/compressedDepth/parameter_descriptions
/head_xtion/rgb/image_color/compressedDepth/parameter_updates
/head_xtion/rgb/image_mono/compressed
/head_xtion/rgb/image_mono/compressed/parameter_descriptions
/head_xtion/rgb/image_mono/compressed/parameter_updates
/head_xtion/rgb/image_mono/compressedDepth
/head_xtion/rgb/image_mono/compressedDepth/parameter_descriptions
/head_xtion/rgb/image_mono/compressedDepth/parameter_updates
/head_xtion/rgb/image_rect/compressed
/head_xtion/rgb/image_rect/compressed/parameter_descriptions
/head_xtion/rgb/image_rect/compressed/parameter_updates
/head_xtion/rgb/image_rect/compressedDepth
/head_xtion/rgb/image_rect/compressedDepth/parameter_descriptions
/head_xtion/rgb/image_rect/compressedDepth/parameter_updates
/head_xtion/rgb/image_rect_color/compressed
/head_xtion/rgb/image_rect_color/compressed/parameter_descriptions
/head_xtion/rgb/image_rect_color/compressed/parameter_updates
/head_xtion/rgb/image_rect_color/compressedDepth
/head_xtion/rgb/image_rect_color/compressedDepth/parameter_descriptions
/head_xtion/rgb/image_rect_color/compressedDepth/parameter_updates

[catkinized_libav] Get both install and devel build to work

Atm only the install build works. The devel build works if you remove the exported libraries but then the install build doesn't doesn't.

Jenkins reported broken build

with the following problem:

In file included from <http://marcs-pc:8080/job/catkin_make_strands_desktop_full/ws/catkin_ws/src/data_compression/openni_saver/src/cv_saver.cpp>:1:0:
<http://marcs-pc:8080/job/catkin_make_strands_desktop_full/ws/catkin_ws/src/data_compression/openni_saver/src/cv_saver.h>:8:45: fatal error: openni_saver/StartStopRecording.h: No such file or directory
compilation terminated.

Please assign somebody to fix this problem or close this issue if it's already fixed.

Hydro compile

Trying to compile a strands install using the Hydro instructions on a fresh install of Ubuntu 12.04, I get this error:

[  9%] Building CXX object data_compression/openni_saver/CMakeFiles/openni_image_saver.dir/src/openni_image_saver.cpp.o
In file included from /opt/strands/strands_catkin_ws/src/data_compression/openni_saver/src/openni_image_saver.cpp:1:0:
/opt/strands/strands_catkin_ws/src/data_compression/openni_saver/src/openni_image_saver.h:7:41: fatal error: openni_saver/LoggingService.h: No such file or directory
compilation terminated.
make[2]: *** [data_compression/openni_saver/CMakeFiles/openni_image_saver.dir/src/openni_image_saver.cpp.o] Error 1
make[1]: *** [data_compression/openni_saver/CMakeFiles/openni_image_saver.dir/all] Error 2
make: *** [all] Error 2
Invoking "make" failed

Not 100% sure it is specifically because of the hydro change...

Remove repo ahead of release

This package is pretty outdated and has not been used. The idea was to be able to compress rosbags more efficiently but with this technique only one camera is supported. At KTH we are currently only doing observations at waypoints and then uploading it to an ftp server. Since the plan otherwise is to trigger complete data collection when needed, I propose to delete this repo since heavy refactoring is needed to get it to do what we want. If needed, my repo can later be merged into a new project.

Your thoughts, @hawesie , @marc-hanheide ?

Dependencies in CMakeLists.txt and package.xml

@nilsbore After I commented on the commit with the include directory I noticed that you specify your dependencies in a quite unusual way. You might want to think of using find_package for opencv and cv_bridge: e.g. find_package(OpenCV REQUIRED) find_package(catkin REQUIRED cv_bridge). And also define it as a build and run time dependecy in the package.xml instead of using absolute paths for linking and includes.

Use regular expressions to exlude image topics when recording rosbag

Right now, strandsbag record is just recording rosout and rosout_agg. Rosbag has functionality for recording everything except stuff that matches regular expressions. Since we will have multiple cameras, we need to know the formatting for these different topics.

Investigate if theora and h264 compression in libav

The theora rgb image compression in ROS is not usable atm for unclear reasons, sometimes it works, sometimes it doesn't. But the FFV1 codec from libav works fine for the depth. So we should investigate if we can use the same image_transport code that we have already written but for the rgb image. Possible codecs could be theora or h264.

Decompress new video frames as needed

Right now, strandsbag play requires you to first decompress the video created by strandsbag record with the libav_compressor video_loader tool. For large collections this would take much time and space. Instead it would be nice to just decompress the videos as new images are required. This could be done by sending a message from bag_player, for instance. The decompressing code can be found in batch_decompressor.py.

Use ffv1 codec in c++ api, to encode file in mov container.

Use option to use the mov container together with the ffv1 codec in the c++ wrappers.

Put this in a properly named repository

When we have closed these issues, this stuff should be in one of the other repositories I guess. Does anyone have any suggestions?

specifing the date when replaying a recorded video

When using the mongodb_play.py utility to replay the rgb and depth topics, is it possible to specify date (datetime) from which the playback should start?
Thanks

Play back openni topics from video together with rosbag

Play back openni topics from video together with rosbag. This also includes timestamping the video frames by storing the times in a separate text file.

Add ros node to do the compression directly from the openni_camera node.

This would be a nice addition, if we do this, would also be nice to turn it into a catkin package. But this is not interesting before we can compress 16 bit depth using ffv1 and the mov container.

[libav_image_transport] apt-get install not linking catkinized_libav

When I install the libav_image_transport on the robot via apt-get, it crashes because /opt/ros/hydro/lib/libav_image_transport is not linking any catkinized_libav libraries. However, when I install it by first downloading, compiling and installing the package, it works fine, everything links correctly. This is also true on my computer. Also on both mine and Rares' machine, the apt-get package actually works. But on Rares' machine /opt/ros/hydro/lib/libav_image_transport does NOT seem to be linking the libraries. @marc-hanheide Do you have any idea what might be going on?

Can't compress grayscale video.

I didn't figure out how to compress grayscale video using only one channel in this API. Add an option to do this in the c++ wrappers.

indigo release?

Hello, could this package be released for indigo?
Thanks you

ping: @marc-hanheide

openni_saver build fails.

Just updated my stable installation and tried to build it:

CMake Error at data_compression/openni_saver/CMakeLists.txt:78 (add_executable):
  Cannot find source file:

    src/synch_saver_node.cpp

[catkinized_libav] Add proper install targets

I'm working on installing catkinized_libav correctly (enabling catkin_make install). I am then copying the include folders and library files to the correct directories in the Cmake file:

https://github.com/nilsbore/data_compression/blob/hydro-devel/catkinized_libav/CMakeLists.txt

But I can't figure out how to also copy the symlinks, or rather the solutions seem overly complicated. Is there any more correct way of doing this? @marc-hanheide , @cdondrup ?