split::words alternative that keeps punctuation

[a-merezhanyi]

split::words alternative that keeps punctuation

[a-merezhanyi]

Do you really need voca in this case? Doesn't split_whitespace work for you?

let s = "Some sort, of a - sentence."
        .split_whitespace()
        .collect::<Vec<&str>>();
assert_eq!(s, ["Some", "sort,", "of", "a", "-", "sentence."]);

[mainrs]

It does in almost all the cases. But I do have similar input to the one you showcased for split::words like XmlHttpRequest. Maybe I have to put something custom made together!

[a-merezhanyi]

Could you please provide a test case to review, so I'd be able to think about the solution?

[a-merezhanyi]

@sirwindfield Please consider the following code snippet, whether this solution works for you?

let s = "Some sort, of a - sentence using LazyLoad with XMLHttpRequest."
    .split_whitespace()
    .collect::<Vec<_>>();
let mut res = Vec::new();
for w in s {
    if w._words().len() > 1 {
        let vec = w._words();
        for v in vec {
            res.push(v);
        }
    } else {
        res.push(w);
    };
}
assert_eq!(
    res,
    [
        "Some", "sort,", "of", "a", "-", "sentence", "using", "Lazy", "Load", "with", "XML",
        "Http", "Request"
    ]
);

Please note that it drops the last point because of words() function behavior; you need to cover such edge cases.

Another possible solution is to cut out the split::words function and to rewrite it for your needs.

[a-merezhanyi]

Closed as it's stalled (workaround has been proposed).