.extend significantly outperforms concat for the same operation

[Chuck321123]

.extend significantly outperforms concat for the same operation

[s-banach]

It's optimized for the use-case where you call df = pl.concat(...) just once up top,
and then you do all the rest of your processing on df.
The idea is that most operations on df will be faster if df is contiguous in memory, rather than separate chunks.

[s-banach]

Can you try concat with rechunk=False?

[Chuck321123]

@s-banach It was about 3000x faster. Are there any reasons this isnt set to False by default?

[Chuck321123]

@s-banach I see. Unfortunately, there were no performance gains when i did rechunk=False with how="align"

[ritchie46]

You are comparing apples with peaches.

Every time you extend df1, you mutate the original memory. So the first extend is much cheaper than the later ones (that trigger a realloc) as your size of df1 increases every iteration.

for i in range(10):
    df1.extend(df2)
    print(df1.height)

1000000
1500000
2000000
2500000
3000000
3500000
4000000
4500000
5000000
5500000

So your extend "benchmark" has a different input every iterations, whilst your concat "benchmark" needs to concat the 5_500_000 case every time. Please take the time to validate your benchmarks.

On the difference, they will perform different. That's fine, they do different things.

Don't use extend as it mutates underlying memory which leads to these kind of bugs. As the warning of the extend docstring states.