daohu527/cyber_record

cyber_record offline parse tool

I am daohu527

• Working : Autonomous vehicles 🚙
• Hobbies : Books 📚 | Running 🏃
• Fun fact : In fact, each of our lives is a world, and even the most ordinary people must fight for the world in which they live.

Discuss

• Interest group : we have a discussion group apolloauto-disscuss. Here we can discuss anything about autonomous driving
• Online talk : We have tech talks every Saturday from 9-10am(beijing time) #TencentConference: 977-2425-1086

Plan

This is my latest development plan

If you have any needs for autonomous driving projects, please feel free to contact us. We have a very mature team that can provide hardware, software and consulting services. email:[email protected]

cyber_record's Issues

Need Help when reading the `.record` file

Hi,

I met a problem when reading the record file.

OS runtime specification:

ubuntu 18.04
python 3.8

Here is the steps to reproduce:

1. Here, demo_3.5.record is the file downloaded from https://apollo-system.cdn.bcebos.com/dataset/6.0_edu/demo_3.5.record
2. write a test file:

from cyber_record.record import Record

file_name = "./demo_3.5.record"
record = Record(file_name)
for topic, message, t in record.read_messages("/apollo/canbus/chassis_detail"):
    print("{}, {}, {}".format(topic, type(message), t))

3. output (no output):

[no output]

4. another test file:

from cyber_record.record import Record

file_name = "./demo_3.5.record"
record = Record(file_name)
for topic, message, t in record.read_messages():
    print("{}, {}, {}".format(topic, type(message), t))

5. also no output
6. using command line:

cyber_record info -f ./demo_3.5.record

7. output (right and appropriate):

record_file: ./demo_3.5.record
version:     1.0
begin_time:  2019-01-07 11:12:57.338835
end_time:    2019-01-07 11:13:42.886741
duration:    45.55 s
size:        91.79 MByte
message_number: 61615
channel_number: 32

/tf_static                            , apollo.transform.TransformStampeds    , 0
/apollo/monitor                       , apollo.common.monitor.MonitorMessage  , 9
/apollo/localization/pose             , apollo.localization.LocalizationEstimate, 4558
/apollo/perception/obstacles          , apollo.perception.PerceptionObstacles , 455
/apollo/planning                      , apollo.planning.ADCTrajectory         , 455
/apollo/sensor/gnss/imu               , apollo.drivers.gnss.Imu               , 4553
/apollo/canbus/chassis_detail         , apollo.canbus.ChassisDetail           , 0
/apollo/hmi/status                    , apollo.dreamview.HMIStatus            , 27
/apollo/routing_request               , apollo.routing.RoutingRequest         , 0
/apollo/sensor/gnss/rtk_eph           , apollo.drivers.gnss.GnssEphemeris     , 132
/tf                                   , apollo.transform.TransformStampeds    , 9079
/apollo/control                       , apollo.control.ControlCommand         , 4539
/apollo/monitor/system_status         , apollo.monitor.SystemStatus           , 9
/apollo/prediction                    , apollo.prediction.PredictionObstacles , 453
/apollo/sensor/gnss/odometry          , apollo.localization.Gps               , 4537
/apollo/navigation                    , apollo.relative_map.NavigationInfo    , 0
/apollo/localization/msf_gnss         , apollo.localization.LocalizationEstimate, 45
/apollo/drive_event                   , apollo.common.DriveEvent              , 0
/apollo/routing_response              , apollo.routing.RoutingResponse        , 0
/apollo/canbus/chassis                , apollo.canbus.Chassis                 , 4536
/apollo/localization/msf_status       , apollo.localization.LocalizationStatus, 4538
/apollo/sensor/gnss/best_pose         , apollo.drivers.gnss.GnssBestPose      , 45
/apollo/sensor/gnss/gnss_status       , apollo.drivers.gnss.GnssStatus        , 45
/apollo/sensor/gnss/corrected_imu     , apollo.localization.CorrectedImu      , 4536
/apollo/sensor/gnss/ins_stat          , apollo.drivers.gnss.InsStat           , 45
/apollo/sensor/gnss/rtk_obs           , apollo.drivers.gnss.EpochObservation  , 92
/apollo/sensor/gnss/raw_data          , apollo.drivers.gnss.RawData           , 13921
/apollo/perception/traffic_light      , apollo.perception.TrafficLightDetection, 291
/apollo/localization/msf_lidar        , apollo.localization.LocalizationEstimate, 180
/apollo/control/pad                   , apollo.control.PadMessage             , 0
/apollo/guardian                      , apollo.guardian.GuardianCommand       , 4535
/apollo/hmi/audio_capture             , apollo.dreamview.AudioCapture         , 0

8. using command line to read specific topic:

cyber_record echo -f ./demo_3.5.record -t /apollo/canbus/chassis_detail

9. no output either

I wonder if I am misunderstanding how to use your tool.

Please let me know if there is any additional information or resources that I can provide to assist with the investigation. I am willing to collaborate and support your team in any way possible.

Add encryptor

Add encryptor for record

fix pylint

run pylint

bash scripts/lint/pylint.sh cyber_record

There are various lint errors that need to be fixed

Invalid proto descriptor for file "": : Missing field: FileDescriptorProto.name.

Hello, I am using this tool to try to parse the demo record file (demo_sensor_data_for_vision.record), and I only modified the related file name in the test_parser_message.py. Now I encountered this issue (I am calling the python code inside the apollo container):

By the way, I also encountered the command not found issue by directly calling cyber_record from the command line in container, and I am using Ubuntu 20.04.

This is a great tool. Anyway how to solve these issues?

cyber_record echo add print out messages proto file

use below cmd to print out message's proto file

cyber_record echo -f file -m topic

无法解析radar数据

目前无法解析radar数据，提示“Couldn't find base/device_connect/proto/radar_data.proto”

TypeError: PointCloudBuilder.build() takes from 3 to 4 positional arguments but 5 were given

Hi,

I'm using radiate dataset, which has csv file for lidar point cloud. The csv file has (x, y, z, intensity, ring) data. Here I'm trying to create a record file for Apollo env. When I feed this csv file, I'm getting an error as TypeError: PointCloudBuilder.build() takes from 3 to 4 positional arguments but 5 were given.

I have also written a function for building from csv in the PointCloudBuilder() but still getting this error

Building from CSV

def build_from_csv(self, csv_file_name, frame_id, t=None):
pb_point_cloud = pointcloud_pb2.PointCloud()

    if t is None:
        t = time.time()

    self._build_header(pb_point_cloud.header, t=t, frame_id=frame_id)
    pb_point_cloud.frame_id = frame_id
    pb_point_cloud.measurement_time = t

    # Open and read the CSV file
    with open(csv_file_name, 'r') as csv_file:
        csv_reader = csv.reader(csv_file)
        next(csv_reader)  # Skip header row if present

        for row in csv_reader:
            x, y, z, intensity = map(float, row)  # Assuming CSV columns are in this order
            point = pb_point_cloud.point.add()
            point.x = x
            point.y = y
            point.z = z
            point.intensity = intensity

    self._sequence_num += 1
    return pb_point_cloud

This function is implemented in the dataset converter for radiate dataset for record to run on Apollo

Implemented in dataset converter

  elif 'lidar_bev_image' in c:
    pb_msg = pc_builder.build_from_csv(f2, id, t/1e9, LIDAR_TRANSFORM) ##This is where I'm getting that error
    channel_name = "/apollo/sensor/{}/compensator/PointCloud2".format(c)
    print("Lidar") 

What changes can I do? Or what is causing this error?

AttributeError: 'Record' object has no attribute '_write_file_header_record'

运行test_write_record.py报错
AttributeError: 'Record' object has no attribute '_write_file_header_record'

write image to the record failed: AttributeError: 'NoneType' object has no attribute 'shape'

Hi @daohu527 ,

I have successfully used your recover tool to repair the file (after updating the cyber_record). Then I split the file to delete the original imaes. My idea is by transforming the image extracted and then writing it back to the record file to test Apollo. However, when I try to write image data back to the record file, it gives me error: AttributeError: 'NoneType' object has no attribute 'shape'

Moreover, I tried to write other images to an empty file, and this error is still encountered, but I have not encountered it before I updated the tool (I am using the same data as before). Is there anything conflict among these updates?

add requirements

Add dependencies for libraries

Error about cyber_record.record

Hello!
Record from cyber_record.record can't work when the 'xxx.record' was obtained by the Apollo-edu version. The error is: Conflict register for file "modules/common/proto/error_code.proto": apollo.common.StatusPb is already defined in file "modules/common_msgs/basic_msgs/error_code.proto". Please fix the conflict by adding package name on the proto file, or use different name for the duplication.

But it can work perfectly when the 'xxx.record' was obtained by the Apollo 7.0 version.

How can I solve this problem? Think you very much!

Add docs

Add documentation and hosting to readthedocs

How to write a muiltiple channels recoder?

My codes like this:
with Record("xxx/output.record", mode='a') as w_record:
w_record.write('/ins/gps', GPS_, int(stamp))
w_record.write('/ins/imu', IMU_, int(stamp))
and I can this see these messsges in recoder info. But It wont work ,when I play this bag in Apollo.
Only first item showed in monitor .

Environment test or provide docker

add unitest

here's doc talk about python test

https://realpython.com/python-testing/#choosing-a-test-runner

api docs

generate api docs

error with protobuf install dependency version

I tested with protobuf 3.19.0, but the setup.py need version >= 3.20.0. which will cause install error!!!

I tested above 3.17.0 will work, so I will update the version dependency to >=3.17.0

proto_desc dependency

Requires topological sort dependency

add example for read record

add example for read record

cyber_record add map and conf to record header

Write map information and configuration information to the record header, or add a new chrunk

Add write message to record

Add write message to record

can i parse GNSS or imu data?

TypeError: Couldn't build proto file into descriptor pool!

I am using cyber_record to read the Cyber record file.
At the moment, I could use it to read messages from channels when the record file is produced by Apollo in the SimControl mode.
However, instead of using SimControl, when using LGSVL to simulate the scenario (connect it through bridge to Apollo), the Cyber record file cannot be read by cyber_record (although the file could still be playable by official cyber_recorder tool).

The error happened when reading the file is as follows:

Traceback (most recent call last):
  File "/sample_dir/test_analyzer.py", line 9, in <module>
    record = Record(record_path)
  File "/sample_dir/venv/lib/python3.7/site-packages/cyber_record/record.py", line 102, in __init__
    self._open(f, mode, allow_unindexed)
  File "/sample_dir/venv/lib/python3.7/site-packages/cyber_record/record.py", line 251, in _open
    if mode == 'r': self._open_read(f)
  File "/sample_dir/venv/lib/python3.7/site-packages/cyber_record/record.py", line 272, in _open_read
    self._reader.start_reading()
  File "/sample_dir/venv/lib/python3.7/site-packages/cyber_record/reader.py", line 74, in start_reading
    self._create_message_type_pool()
  File "/sample_dir/venv/lib/python3.7/site-packages/cyber_record/reader.py", line 240, in _create_message_type_pool
    self._add_dependency(proto_desc)
  File "/sample_dir/venv/lib/python3.7/site-packages/cyber_record/reader.py", line 233, in _add_dependency
    self.desc_pool.Add(file_desc_proto)
TypeError: Couldn't build proto file into descriptor pool!
Invalid proto descriptor for file "modules/perception/proto/traffic_light_detection.proto":
  modules/perception/proto/traffic_light_detection.proto: A file with this name is already in the pool.

Given record file for reproducing this bug: https://drive.google.com/file/d/1qr_4Jhz0KWrXsdgcGVdrvsfQ65GB0nCq/view?usp=sharing

Incorrect construction of RecordException

In case of broken .record file there is an attempt of creating RecordException object:

cyber_record/cyber_record/reader.py

Line 268 in 83656db

raise RecordException('expecting {} bytes, read {}'.format(size, len(data)))

but it has overrided init function without accepting additional arguments:

cyber_record/cyber_record/record_exception.py

Line 19 in 83656db

def __init__(self) -> None:

I suggest, that RecordException class should be left empty:

class RecordException(Exception):
    pass

cyber_record：未找到命令(command not found)

After pip3 install cyber_record, I run the following command in my terminal:
cyber_record info -f 0818_test_data/20220818160638.record.00000

And get the "cyber_record：未找到命令(command not found)" error.

What happened?

Remove print(header) from reader or use a flag to control

Hi!

Thank you for the great tool!

I am wondering if the following line

cyber_record/cyber_record/reader.py

Line 56 in b00546d

print(header)

should be removed or controllable through a boolean flag, so it does not print information to the console if I do not need it.

Add cyber_recorder command support

Features to be supported

1. cyber_record info
2. cyber_record play
3. cyber_record record
4. cyber_record split
5. cyber_record recover
6. cyber_record echo

ref

• http://wiki.ros.org/rosbag/Commandline
• http://wiki.ros.org/rostopic

add statistics info

1. Add frequency statistics for each message
2. Add message alignment statistics

Can not import Record from cyber_record.record

Hi, when try to import Record from cyber_record.record, the record_pb2.py throw an error:

module "google.protobuf.descriptor" has no attribute '_internal_create_key'

would you give some advice?

Thanks,

change tests to test

According to the packaging rules of python url. We need to modify the test directory and the file names in it

Add compression support

add record compression support

add parsing message example

add parsing message example

how to add custom message type

Hi, thanks for your great work on the offline cyber record. I want to add some custom message type, how can I make it work?

add recover cmd to recover record file

1. protoc --include_imports --descriptor_set_out=file modules/common_msgs/sensor_msgs/sensor_image.proto
2. descriptor_pb2.FileDescriptorSet.ParseFromString()

add mode='a' for record

Increase speed, show progress bar

Increase speed, show progress bar

Add query options

Add query filter