pit30m/pit30m

The Python SDK for the Pit30M large scale visual localization dataset.

Pit30M Development Kit

Overview

This is the Python software development kit for the Pit30M benchmark for large-scale global localization. The devkit is currently in a pre-release state and many features are coming soon!

Consider checking out the original paper. If you would like to, you could also follow some of the authors' social media, e.g., Julieta's or Andrei's in order to be among the first to hear of any updates!

Getting Started

The recommended way to interact with the dataset is with the pip package, which you can install with:

pip install pit30m

The devkit lets you efficiently access data on the fly. Here is a "hello world" command which renders a demo video from a random log segment. Note that it assumes ffmpeg is installed:

python -m pit30m.cli multicam_demo --out-dir .

To preview data more interactively, check out the tutorial notebook.

More tutorials coming soon.

Torch Data Loading

We provide basic log-based PyTorch dataloaders. Visual-localization-specific ones are coming soon. To see an example on how to use one of these dataloaders, have a look at demo_dataloader in torch/dataset.py.

An example command:

python -m pit30m.torch.dataset --root-uri s3://pit30m/ --logs 00682fa6-2183-4a0d-dcfe-bc38c448090f,021286dc-5fe5-445f-e5fa-f875f2eb3c57,1c915eda-c18a-46d5-e1ec-e4f624605ff0 --num-workers 16 --batch-size 64

Features

• Framework-agnostic multiprocessing-safe log reader objects
• PyTorch dataloaders

In-progress

• More lightweight package with fewer dependencies.
• Very efficient native S3 support through AWS-authored PyTorch-optimized S3 DataPipes.
• Support for non-S3 data sources, for users who wish to copy the dataset, or parts of it, to their own storage.
• Tons of examples and tutorials. See examples/ for more information.

Development

Package development, testing, and releasing is performed with poetry. If you just want to use the pit30m package, you don't need to care about this section; just have a look at "Getting Started" above!

1. Install poetry
2. Setup/update your dev virtual environments with poetry install --with=dev in the project root
   • If you encounter strange keyring/credential errors, you may need PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring poetry install --with=dev
3. Develop away
   • run commands like poetry run python -m pit30m.cli
4. Test with poetry run pytest
   • Advanced command: poetry run pytest -ff --new-first --quiet --color=yes --maxfail=3 -n 4
   • This command will run tests, then wait for new changes and test them automatically. Test execution will run in parallel thanks to the -n 4 argument.
   • The command lets you get feedback on whether your code change fixed or broke a particular test within seconds.
5. Make sure you lint. Refer to ci.yaml for exact commands we run in CI.

• format with black: poetry run black .
• type check with mypy: poetry run mypy . --ignore-missing-imports

6. Remember to run poetry install after pulling and/or updating dependencies.

Note that in the pre-release time, torch will be a "dev" dependency, since it's necessary for all tests to pass.

Publishing

First Time

1. Configure poetry with a PyPI account which has access to edit the package. You need to make sure poetry is configured with your API key.

New Release

1. Decide on the commit with the desired changes.
2. Bump the version number in pyproject.toml.
3. git tag vA.B.C and git push origin <tag>
4. poetry publish --build will build and upload the package to PyPI.
5. Create the release on GitHub & upload the newly generated package files from the previous step with it.

Citation

@inproceedings{martinez2020pit30m,
  title={Pit30m: A benchmark for global localization in the age of self-driving cars},
  author={Martinez, Julieta and Doubov, Sasha and Fan, Jack and B{\^a}rsan, Ioan Andrei and Wang, Shenlong and M{\'a}ttyus, Gell{\'e}rt and Urtasun, Raquel},
  booktitle={{IROS}},
  pages={4477--4484},
  year={2020},
  organization={IEEE}
}

Acknowledgments

There are tons of additional people whose support helped this project happen. Thank you Yevgeni Litvin, Nemanja Djuric, Su Zhaoen, and Carl Wellington for supporting the dataset release. Your support is truly appreciated! Thanks to Thomas Fähse for his detailed blog post on how to effectively and efficiently anonymize automotive data.

Further thanks to the wonderful folks at AWS who supported the release of the Aurora Multi-Sensor Dataset as part of the AWS Open Data Sponsorship Program: Matt Williams, Joe Flasher, Mike Jeffe, and Peter Schmiedeskamp. Thank you for helping make open data accessible to everyone!

Additional Details

Images

Compression

The images in the dataset are stored using lossy WebP compression at quality level 85. We picked this as a sweet spot between space- and network-bandwidth-saving (about 10x smaller than equivalent PNGs) and maintaining very good image quality for tasks such as SLAM, 3D reconstruction, and visual localization. The images were saved using Pillow 9.2.0.

The s3://pit30m/raw prefix contains lossless image data for a small subset of the logs present in the bucket root. This can be used as a reference by those curious in understanding which artifacts are induced by the lossy compression, and which are inherent in the raw data.

Known Issues

A fraction of the images in the dataset exhibit artifacts such as a strong purple tint or missing data (white images). An even smaller fraction of these purple images sometimes shows strong blocky compression artifacts. These represent a known (and, at this scale, difficult to avoid) problem; it was already present in the original raw logs from which we generated the public facing benchmark. Perfectly blank images can be detected quite reliably in a data loader or ETL script by checking whether np.mean(img) > 250.

On example of a log with many blank (whiteout) images is 8438b1ba-44e2-4456-f83b-207351a99865.

Ground Truth Sanitization

Poses belonging to test-query are not available and have been removed with zeroes / NaNs / blank submap IDs in the corresponding pose files and indexes. The long term plan is to use this held-out test query ground truth for a public leaderboard. More information will come in the second half of 2023. In the meantime, there should be a ton of data to iterate on using the publicly available train and val splits.

pit30m's Issues

Missing front wide image indices

@AndreiBarsan Could you please double check

MISSING = {
    UUID("327b7948-4e5f-4d8c-f08b-4fc44a067996"),
    UUID("c9e9e7a7-f1cb-4af8-c5c9-3a610cbcc20e"),
}

I get a

ValueError: Index file not found: s3://pit30m/ {UUID} /cameras/hdcam_12_middle_front_roof_wide/index/index_v0.npz!

error when I try to load the index files for camera MIDDLE_FRONT_WIDE.

It seems like other logs without images return an empty array, which I think is better in terms of error handling.

Noisy warnings in Torch example

Running the torch data loader example produces lots of (expected, I believe) warnings:

poetry run python -m pit30m.torch.dataset --root-uri s3://pit30m/ --logs 00682fa6-2183-4a0d-dcfe-bc38c448090f,021286dc-5fe5-445f-e5fa-f875f2eb3c57,1c915eda-c18a-46d5-e1ec-e4f624605ff0 --num-workers 16 --batch-size 64
root_uri s3://pit30m/
logs: ['00682fa6-2183-4a0d-dcfe-bc38c448090f', '021286dc-5fe5-445f-e5fa-f875f2eb3c57', '1c915eda-c18a-46d5-e1ec-e4f624605ff0']                                                     
WARNING: There are 46 / 23041 (0.20%) timestamp associations with gap > 0.500s. Max is 2.826s
WARNING: There are 608 / 16626 (3.66%) timestamp associations with gap > 0.500s. Max is 29.139s

We should update the code to skip them.

Double check `009334fa-acb9-4813-c60c-1916c83974c8` raw wgs84 poses

wgs84.npz.lz4 does not seem to exist in the bucket. Could you please double check @AndreiBarsan?

If it does not exist, I can submit a small PR that hardcodes this in the MonkeyWrench / LogReader.

Also:

01e86404-c224-4b10-d12e-9e2144626b3b

PyTorch dataloader can read splits, not just raw logs

Support wgs84 value in sensor indices

Would simplify the logic of partitions, and would probably be helpful elsewhere. It can be computed on the fly.

Request: Make the sensor time field of indices uniform

Right now, it is called "img_time" for cameras, and "lidar_time" for lidar. Some of the internal logic would be cleaner if these two fields were simply called "sensor_time" or something like that

add partitioning logic

Move it from the internal repo so other people who may find it useful can find it and re-use it

Off-place frames

I have made a video for the frontal camera of log e9511854-f657-47bd-c9d3-047187cfc663. Watching the video, I observed some frames that "jump", out of place. For a concrete example, consider the following.

Dumping the images shows that frame 5063 looks out place:

And this is what frames 5062-5064 look like:

On the upper left, you can see the img_time (converted to UTC), and the name of the file. As you can see, the time is continuous, but the frame is out of place. Note that the frames have been sorted by img_time

pit30m/pit30m/data/log_reader.py

Line 337 in d4b9ab3

sort_by: str = "img_time",

Support submap gaps when indexing in order to cover the two currently unindexable logs

Convert everything to UNIX or GPS timestamps

Since I have to match things on the fly to filter indices by partitions, I need the timestamps to be comparable. Right now, it seems that indices (v 1 and above) have UNIX timestamps, but raw poses have GPS timestamps, so I have to either convert the raw timestamps to UNIX, or convert the index poses back to GPS to make them comparable. This bit me and I think I was only able to figure it out because I reviewed some indexing-specific PRs. I think we should either move everything to UNIX, or leave it in GPS format. What do you think @AndreiBarsan?

Data cleaning epic

• Delete v0 indexes (safe once NetVLAD prototype validated)
• Delete v1 indexes (safe once NetVLAD prototype validated)
• Delete old LiDAR (once optimized LiDAR in place)

Visualizations

Reproduce visualizations from the paper

• Partitions
• Sun vs time of day over year
• kepler.gl files

Tutorials and examples v1.0

Orthogonal to package updates (e.g., better dataloaders, splits, etc.) we also want to keep improving the available demos, cookbook entries, and tutorials.

• ❗ Update tutorial with latest log reader format, clean up your explanations, etc.
• COLMAP prototype working locally
• COLMAP demo notebook
• Multi-sensor demo which accounts for async LiDAR (transform sweeps to exact camera times using pose interpolation)
• “Open in SageMaker Studio” buttons for all tutorials in the devkit README
  • Important: Highlight how the users don’t need to s3 sync the data to do any of this! (Eg in contrast with Boreas where all instructions start with “download the data from S3 to some local folder”)
  • Follow ideas from this AWS example for Boreas & credit Keenan's awesome work: https://github.com/utiasASRL/pyboreas/tree/master/pyboreas/tutorials/aws
• Visualize calibration (open3d coordinate frames wrt vehicle origin)

v0.2.0

Goal: mid-June August 2023

• support pose interpolation for time-accurate LiDAR-camera alignment
• Tool to merge indexes for supporting multi-camera iteration
  • Now we need to account for differences in timing between logs, since some will have global shutter but some won’t, in which case how we index is a matter of user preference.
• Finish indexing all data jointly
• Support multi-sensor indexes in devkit
• Add pre-computed semantic segmentation
• Add weather metadata to devkit
• Update ETL or index code to extract raw data into non-numpy stuff, at least for poses and calibration
• Update Pit30M arXiv to point to new homepage at pit30m.github.io
• Document exact compression arguments
• Document ‘raw’ data existence as a bonus
• Add GitHub-friendly citation file to repo like Argoverse 2

Stretch goals:

• Tool to easily analyze % of un-indexed or un-indexable data (e.g., camera images which can’t be matched to a log) for reporting to users - what is the gap going from the 30M in pit30m to the >50M samples we actually have in the raw dataset?
• Tool to merge indexes across logs, so I can, e.g., geo-query Pit30M data and get results from all matching logs

v0.0.2 indexing improvements

• #16
• #22
• #44
• Store UTM altitude in indexes
• Hook up index sanitization logic to index generation (no-op - if you create an index from sanitized poses you will get sanitized data in the indexes)
• Put UNIX timestamps in the indexes, no GPS
• Ensure indexes are sorted before dumping the final result
• Switch everything back to GPS for simplicity
• Allow (re-)indexing to run on AWS

Not really necessary upon further investigation:

• Allow indexing code to read original pose.npz files, including those which may have sensitive data, i.e., not just whatever is in the public bucket

Automate benchmarking of LogReader methods

As we keep adding more functionality, some methods are becoming very slow, and we do currently not have a good way to keep track of this.

bundle splits with dataset

This just means that there is a method to load the partitions from the devkit. This could be done with text files in the repo itself, or putting them in the AWS bucket.

Reproduce NetVLAD baseline

Reproduce and show basic recall@N numbers for off-the-shelf netvlad features, from the front camera only

Unbreak tutorial_00_introduction

• Make sure it runs
• Use indices instead of iterators

v0.1.0

• #29
• make torch dependency fully optional
• integrate calibration support in log reader
  • nice API to transform points between sensors
  • no more GPS time, e.g., in LiDAR point times (we switched back to GPS everywhere for consistency)
  • preprocess raw_calibration.yml
• specify exact PyPI release process via GitHub releases with proper tages, etc.
• log-level download functionality in the CLI
• improve README with a teaser image and explain dataloader vs. cli
• add acknowledgements to README
• create tasks for integrating with other datasets/toolkits

Reproduce ResNet50 baselines

For both front camera and lidar

Ensure sensor indices are sorted by sensor timestamp

How to access IMU/GPS data?

Hi,

Thank you for the dataset! I'm not sure how should I go about getting the IMU/GPS data from the dataset. I have dug around and had some questions:

• Regarding GPS, is the correct function to access them raw_wgs84_poses_dense?

• Regarding the IMU, I tried the following (follow what monkeywrench does)

root = "s3://pit30m/"
log_root_uri=os.path.join(root, sample_log_id)

fs = fsspec.filesystem(urlparse(root).scheme)
lr = LogReader(log_root_uri=log_root_uri)

with fs.open(os.path.join(log_root_uri, "raw_imu.npz.lz4"), "rb") as raw_f:
    with lz4.frame.open(raw_f, "rb") as f:
        raw_imu = np.load(f, allow_pickle=True, encoding="latin1")["data"]
        print(raw_imu)

However, it throws this error:

NoCredentialsError: Unable to locate credentials

Any help would be much appreciated!

Add `__version__` to released packages

Our package root needs a __version__ member that stores a package's version as it's a de facto python standard. Should find a way to automatically update it when creating a release.

code to read splits in the devkit, with tests

LiDAR -> IMU transformation

Hi. How can I access the transformation between different sensors? I couldn't find that information on the website/paper or the codebase. Thanks!

Regenerate LiDAR index to fix path length

Should regen this and then solve Julieta's TODO in get_lidar.

Data optimization epic

Important but non-blocking things to improve in the bucket. That is, improvements which need bucket writes. Of course, this covers known things. Once users start utilizing our dataset, it is likely they will uncover small issues and possible improvements we will also need write access to update.

• #59
• pre-cache a log's camera metadata into single npy file
• optimize calibration files to be more self-contained (right now they are usable but require complicated software work), especially raw_calibration.yml
• clean up and re-upload vehicle data time series (e.g., engine state, etc.)
• upload weather metadata

Clarify data license and access rights in docs

• mention data license in README (CC BY-NC-SA 4.0)
• mention you don't need an AWS account to access anything

HD Map Missing?

Thanks for releasing the dataset.

The dataset contains lidar, camera, and ego-pose data.
However, I don't find the "HD map" data.
I want to know how to get it.

Remove "no GT available" README warning and release 0.0.2

Should be done by EoD Jun 15 ideally.

partitions and splits ready for release, including txt file specifying exact intervals which are secret

@una-dinosauria This is what you are already working on! I am just tracking it here. 😄

My proposal is to define secret intervals as follows:

• an interval centered around each test-query
• we denote the time of this query as t
• the "secret interval" is the union of:
  • (t - 15s, t + 15s), and
  • (pose_t - 30m, pose_t + 30m)

Secret intervals could then be specified as a CSV with the following structure:

log_id,start_timestamp,end_timestamp

This way whenever I build a fresh index version, any row that would fall within that interval would be set to a blank pose. Of course, the pose data to actually build the indexes comes from the private raw pose files.

Let me know what you think!

Create pip package

User should be able to do pip install pit30m and start playing with the dataset.

v0.0.2

CC @una-dinosauria @AndreiBarsan

• #31
• fix torch dataset Map compat
• #27
• #28
• #23
• #26
• upload package to production pypi
• fully deprecate index v0
• re-process data indexes with test-query poses removed

optimize LiDAR to non-redundant vehicle-relative f32

• write, test, and land modernization tool
• process all data
• generate new LiDAR index to point to optimized files
• update devkit to read new format

re-add GT poses where applicable (everywhere except for test-queries and their vicinity)