Expression input not working correctly for either argument of `Expr.str.replace_many`

[henryharbeck]

Expression input not working correctly for either argument of `Expr.str.replace_many`

[cmdlineluser]

Isn't that the reason for its existence?

It's supposed to perform multiple replacements in each string, i.e. as if you did:

df.with_columns(
    pl.col("text")
    .str.replace_many(["123", "abc"], ["888", "zzz"])
    .alias("replaced")
)

# shape: (2, 3)
# ┌────────┬─────────────┬──────────┐
# │ text   ┆ replacement ┆ replaced │
# │ ---    ┆ ---         ┆ ---      │
# │ str    ┆ str         ┆ str      │
# ╞════════╪═════════════╪══════════╡
# │ 123abc ┆ 888         ┆ 888zzz   │
# │ abc456 ┆ zzz         ┆ zzz456   │
# └────────┴─────────────┴──────────┘

(Perhaps some simpler examples could be added to the docs?)

[henryharbeck]

Yes, but what if I wanted to provide a different replacement value for many string patterns per row? Otherwise, it seems kind of pointless to accept expression arguments.

How should I otherwise achieve the expected result?

df = pl.DataFrame({
    "text": ["123abc", "abc456"],
    "replacement": ["888", "zzz"],
})

df.with_columns(
    pl.col("text")
    .str.replace_many(["123", "abc"], pl.col("replacement"))
    .alias("replaced")
)

# Actual
shape: (2, 3)
┌────────┬─────────────┬──────────┐
│ text   ┆ replacement ┆ replaced │
│ ---    ┆ ---         ┆ ---      │
│ str    ┆ str         ┆ str      │
╞════════╪═════════════╪══════════╡
│ 123abc ┆ 888         ┆ 888zzz   │ # <---- note the zzz from the below row
│ abc456 ┆ zzz         ┆ zzz456   │
└────────┴─────────────┴──────────┘

# Expected
shape: (2, 3)
┌────────┬─────────────┬──────────┐
│ text   ┆ replacement ┆ replaced │
│ ---    ┆ ---         ┆ ---      │
│ str    ┆ str         ┆ str      │
╞════════╪═════════════╪══════════╡
│ 123abc ┆ 888         ┆ 888888   │
│ abc456 ┆ zzz         ┆ zzz456   │
└────────┴─────────────┴──────────┘

[cmdlineluser]

Yeah, you want something like:

df.with_columns(
    pl.col("text").str.replace(pl.col("text").head(3), pl.col("replacement"))
      .alias("replaced")
)
# ComputeError: dynamic pattern length in 'str.replace' expressions is not supported yet

• #14367

Whereas replace_many is for doing similar to:

pl.col("text")
  .str.replace("123", "888")
  .str.replace("abc", "zzz")

[henryharbeck]

I don't think I am after pl.col("text").head(3) as I would like to replace both "123" and "abc" on each row.
My apologies if I didn't get that across well.

Like so:

df.with_columns(
    pl.col("text")
    .str.replace_many(["123", "abc"], "888")
    .alias("replaced")
)

shape: (2, 3)
┌────────┬─────────────┬──────────┐
│ text   ┆ replacement ┆ replaced │
│ ---    ┆ ---         ┆ ---      │
│ str    ┆ str         ┆ str      │
╞════════╪═════════════╪══════════╡
│ 123abc ┆ 888         ┆ 888888   │
│ abc456 ┆ zzz         ┆ 888456   │
└────────┴─────────────┴──────────┘

but instead of replacing both of the strings with "888" for every row, I would like the replacement value (for both "123" and "abc") to be based on pl.col("replacement") (i.e. the value in the given row)

[henryharbeck]

To be honest, I just came across this when trying to give expression inputs to Expr.str.replace_many. I don't really need to do this type of replacement, but was more so seeing what was possible. If expressions are supported here, the behaviour definitely seems unexpected to me.

Looking at Series.str.replace_many, expressions are not supported, so perhaps the type signature for Expr.str.replace_many is more generous than what is (currently) supported...? 🤷‍♂️

Same thing goes for str.contains_any with the different signatures between Series and Expr

[deanm0000]

I really don't know if this is how it is intended to work but this gets your intended result (at least for example 1 and 2)

def cust_replace_many(source: pl.Expr|str, patterns:list, replace_with:pl.Expr|str):
    if isinstance(source, str):
        source=pl.col(source)
    if isinstance(replace_with, str):
        replace_with=pl.col(replace_with)
    for x in patterns:
        source=source.str.replace(x, replace_with)
    return source

df.with_columns(replaced=cust_replace_many('text', ['123','abc'], 'replacement'))
shape: (2, 3)
┌────────┬─────────────┬──────────┐
│ text   ┆ replacement ┆ replaced │
│ ---    ┆ ---         ┆ ---      │
│ str    ┆ str         ┆ str      │
╞════════╪═════════════╪══════════╡
│ 123abc ┆ 888         ┆ 888888   │
│ abc456 ┆ zzz         ┆ zzz456   │
└────────┴─────────────┴──────────┘

If you monkey patch it to pl.Expr then it works off of the source column

pl.Expr.cust_replace_many=cust_replace_many
df.with_columns(replaced=pl.col('text').cust_replace_many(['123','abc'], 'replacement'))
shape: (2, 3)
┌────────┬─────────────┬──────────┐
│ text   ┆ replacement ┆ replaced │
│ ---    ┆ ---         ┆ ---      │
│ str    ┆ str         ┆ str      │
╞════════╪═════════════╪══════════╡
│ 123abc ┆ 888         ┆ 888888   │
│ abc456 ┆ zzz         ┆ zzz456   │
└────────┴─────────────┴──────────┘

For example 3 you could do this:

(
    df
    .group_by('replace', maintain_order=True)
    .agg(
        pl.col('text').first(),
        replaced=pl.col('text').str.replace_many('replace','woo').first()
        )
)
shape: (2, 3)
┌─────────┬────────┬──────────┐
│ replace ┆ text   ┆ replaced │
│ ---     ┆ ---    ┆ ---      │
│ str     ┆ str    ┆ str      │
╞═════════╪════════╪══════════╡
│ 123     ┆ 123abc ┆ woowoo   │
│ abc     ┆ abc456 ┆ woo456   │
└─────────┴────────┴──────────┘

Actually the group_by approach works for example 1 too

(
    df
    .group_by('replacement', maintain_order=True)
    .agg(
        pl.col('text').first(),
         replaced=pl.col('text').str.replace_many(["123", "abc"],pl.col("replacement")).first()
         )
)
shape: (2, 3)
┌─────────────┬────────┬──────────┐
│ replacement ┆ text   ┆ replaced │
│ ---         ┆ ---    ┆ ---      │
│ str         ┆ str    ┆ str      │
╞═════════════╪════════╪══════════╡
│ 888         ┆ 123abc ┆ 888zzz   │
│ zzz         ┆ abc456 ┆ zzz456   │
└─────────────┴────────┴──────────┘

[cmdlineluser]

I would like the replacement value to be based on the value in the given row

You can mimic the behaviour with a window expression:

df.with_columns(
    pl.col("text")
    .str.replace_many(["123", "abc"], pl.col("replacement").first())
    .over("replacement")
    .alias("replaced")
)

shape: (2, 3)
┌────────┬─────────────┬──────────┐
│ text   ┆ replacement ┆ replaced │
│ ---    ┆ ---         ┆ ---      │
│ str    ┆ str         ┆ str      │
╞════════╪═════════════╪══════════╡
│ 123abc ┆ 888         ┆ 888888   │
│ abc456 ┆ zzz         ┆ zzz456   │
└────────┴─────────────┴──────────┘

But that specific task is just not what replace_many is "designed" for. (Or perhaps I am mistaken?)

[henryharbeck]

Thank you both for being forthcoming with suggestions.

I will say I am still confused about how expressions are intended to be used (if at all) in Expr.str.replace_many
It does not seem like they work "out of the box" unless the column being referred to only has 1 unique value (at which point you might as well use a string literal)

Example 2 works when a string literal is passed as the 2nd argument instead of an expression.

I am starting to question whether expressions were intended to be supported at all? Or whether the signature was intended to be more like Series.str.replace_many