robmdunn/faster-data-generation

Generating data 100x faster with Rust

Generating data 100x faster with Rust

Example scripts to demonstrate the ability of Rust to accelerate data generation. Performance improvement can vary, initial measurements currently show 100-150x gain for a typical dataset/ configuration in v1.

v2 version is still a work in progress, although most functionality is working it may have some rough edges. Performance with Rayon threading is still something I am exploring, when scaling up it does not saturate all the cores perfectly. If you know why feel free to send me a PR.

PRs on any other performance improvement are welcome, as the goal of this project is to get the fastest data generation possible. While we are currently at 100x -- have a sense we can push this much higher over time.

Versions

v1 (Python / Rust)

• generate dataset

v2 (Python / Rust)

• ability to pass parameters via CLI
• dynamic schema loading (Rust-only)
• enable threading (multi-core)
• convert to dataframe
• export to Parquet

Run examples

Python

# enter directory
cd examples/pyfake-v1

# v1 requires faker
pip install faker
# v2 requires faker and polars
pip install faker polars

# run
python pyfake/generate.py

# benchmark row (average over 10 runs)
python -c 'import pyfake; pyfake.benchmark_row()'

# benchmark column (average over 10 runs)
python -c 'import pyfake; pyfake.benchmark_column()'

Python with Poetry

# enter directory
cd examples/pyfake-v1

# install dependencies
poetry update

# run script
poetry run pyfake

Rust

# enter directory
cd examples/rsfake-v1

cargo build --release
target/release/rsfake

Docker

For convenience a Dockerfile is included with both Python and Rust dependencies pre-installed.

# build
docker build -t fakeroo .

# run interactive shell
docker run -ti  --rm fakeroo bash

# run Python example
cd /examples/pyfake-v1
python pyfake/generate.py

# run Rust example
cd /examples/rsfake-v1
bin/rsfake