tiagoantao/python-performance

Repository for the book Fast Python - published by Manning

Find my homepage at https://tiago.org

Articles

You can find my articles on several topics on my blog

Books

Fast Python - Writing efficient Python code. Repository here

Bioinformatics with Python Cookbook - third edition - Python for Bioinformatics. Repository here

python-performance's Issues

pyarrow parquet schema is defined not completely / correctly

https://tiago.org/yellow_tripdata_2020-01.csv.gz is downloaded into big_data

import pyarrow.parquet as pq
from pyarrow import csv

table = csv.read_csv("../big_data/yellow_tripdata_2020-01.csv.gz")
pq.write_table(table, "../big_data/yellow_tripdata_2020-01.parquet")

Pandas can read the parquet file

import pandas as pd

df = pd.read_parquet("../big_data/yellow_tripdata_2020-01.parquet")
df.head()
#    VendorID tpep_pickup_datetime tpep_dropoff_datetime  passenger_count  trip_distance  RatecodeID store_and_fwd_flag  ...  extra  mta_tax  tip_amount  tolls_amount  improvement_surcharge  total_amount  congestion_surcharge
# 0       1.0  2020-01-01 00:28:15   2020-01-01 00:33:03              1.0            1.2         1.0                  N  ...    3.0      0.5        1.47           0.0                    0.3         11.27                   2.5
# 1       1.0  2020-01-01 00:35:39   2020-01-01 00:43:04              1.0            1.2         1.0                  N  ...    3.0      0.5        1.50           0.0                    0.3         12.30                   2.5
# 2       1.0  2020-01-01 00:47:41   2020-01-01 00:53:52              1.0            0.6         1.0                  N  ...    3.0      0.5        1.00           0.0                    0.3         10.80                   2.5
# 3       1.0  2020-01-01 00:55:23   2020-01-01 01:00:14              1.0            0.8         1.0                  N  ...    0.5      0.5        1.36           0.0                    0.3          8.16                   0.0
# 4       2.0  2020-01-01 00:01:58   2020-01-01 00:04:16              1.0            0.0         1.0                  N  ...    0.5      0.5        0.00           0.0                    0.3          4.80                   0.0
#
# [5 rows x 18 columns]

But the schema is defined not completely / correctly

parquet_file = pq.ParquetFile("../big_data/yellow_tripdata_2020-01.parquet")
print("parquet_file.schema:\n", parquet_file.schema)
# parquet_file.schema:
#  <pyarrow._parquet.ParquetSchema object at 0x708424778300>
# required group field_id=-1 schema {
#   optional int64 field_id=-1 VendorID;
#   optional int64 field_id=-1 tpep_pickup_datetime (Timestamp(isAdjustedToUTC=false, timeUnit=milliseconds, is_from_converted_type=false, force_set_converted_type=false));
#   optional int64 field_id=-1 tpep_dropoff_datetime (Timestamp(isAdjustedToUTC=false, timeUnit=milliseconds, is_from_converted_type=false, force_set_converted_type=false));
#   optional int64 field_id=-1 passenger_count;
#   optional double field_id=-1 trip_distance;
#   optional int64 field_id=-1 RatecodeID;
#   optional binary field_id=-1 store_and_fwd_flag (String);
#   optional int64 field_id=-1 PULocationID;
#   optional int64 field_id=-1 DOLocationID;
#   optional int64 field_id=-1 payment_type;
#   optional double field_id=-1 fare_amount;
#   optional double field_id=-1 extra;
#   optional double field_id=-1 mta_tax;
#   optional double field_id=-1 tip_amount;
#   optional double field_id=-1 tolls_amount;
#   optional double field_id=-1 improvement_surcharge;
#   optional double field_id=-1 total_amount;
#   optional double field_id=-1 congestion_surcharge;
# }

Look at field_id=-1.

Python 3.11
pyarrow 16.0.0
pandas 2.2.2

Profiling code to detect performance bottlenecks link not working

In readme.md file Profiling code to detect performance bottlenecks link returns 404 page as response.

Where is the file 07-pandas/sec1-intro/yellow_tripdata_2020-01.csv.gz?

The file yellow_tripdata_2020-01.csv.gz is missing... At least, I cannot see it.