How to handle "could not map to Unicode" warnings?

[dagolden]

How to handle "could not map to Unicode" warnings?

[kentfredric]

Couldn't tell if this is a new bug, or the same bug:

Can't decode ill-formed UTF-8 octet sequence <E9> in position 5623 at /home/kent/perl5/perlbrew/perls/perl-5.19.5/lib/site_perl/5.19.5/Path/Tiny.pm line 679.
Can't decode ill-formed UTF-8 octet sequence <E9> in position 10096 at /home/kent/perl5/perlbrew/perls/perl-5.19.5/lib/site_perl/5.19.5/Path/Tiny.pm line 679.

This annoying, because its cropping up in the midst of processing several hundred files, and I have no idea which file the warning is pertaining to.

R-ing the source says its not any underlying mechanisms job to report this, because the underlying mechanisms aren't dealing with files at all, they only deal with the bytes.

So it makes sense its Path::Tiny's job to add filename context to this warning, ... though how to do that is anyones guess.

Could you please tell me more about what you're processing: i.e. how and why? I'd like to better understand some use cases before deciding on what Path::Tiny should do.

[kentfredric]

I'm just interating the contents of Dist/Zilla/Plugin/* , oddly enough, seems the problem is Test::Compile, specificially, Test::Compile has a =pod section in ISO-8859-1

https://metacpan.org/source/ETHER/Dist-Zilla-Plugin-Test-Compile-2.037/lib/Dist/Zilla/Plugin/Test/Compile.pm#L198

[chansen]

Unicode::UTF8 supports fallbacks for encode_utf8() and decode_utf8() where you can report any warnings or throw exceptions.

Example:

diff --git a/lib/Path/Tiny.pm b/lib/Path/Tiny.pm
index c914332..32a343a 100644
--- a/lib/Path/Tiny.pm
+++ b/lib/Path/Tiny.pm
@@ -1137,7 +1137,24 @@ sub slurp_raw { $_[1] = { binmode => ":unix" }; goto &slurp }

 sub slurp_utf8 {
     if ( defined($HAS_UU) ? $HAS_UU : $HAS_UU = _check_UU() ) {
-        return Unicode::UTF8::decode_utf8( slurp( $_[0], { binmode => ":unix" } ) );
+        my $path = $_[0]->[PATH];
+        my $fallback = sub {
+            my ($octets, $usv, $position) = @_;
+
+            my $msg;
+            if ($usv) {
+                $msg = sprintf "Can't interchange noncharacter code point U+%X in file '%s' at position %d",
+                    $usv, $path, $position;
+            }
+            else {
+                $msg = sprintf "Can't decode ill-formed UTF-8 octet sequence <%s> in file '%s' at position %d",
+                    join(' ', map { sprintf '%.2X', ord } split //, $octets), $path, $position;
+            }
+            Carp::carp($msg);
+            return "\x{FFFD}";
+        };
+        no warnings 'utf8';
+        return Unicode::UTF8::decode_utf8( slurp( $_[0], { binmode => ":unix" } ), $fallback);
     }
     else {
         $_[1] = { binmode => ":raw:encoding(UTF-8)" };

Would output:

 $ perl -Mlib=lib -MPath::Tiny -wle 'path("~/dev/bad")->slurp_utf8;'
Can't decode ill-formed UTF-8 octet sequence <EF BF> in file '/Users/chansen/dev/bad' at position 4 at -e line 1

[ilmari]

PerlIO::encoding takes its fallback behaviour from the value of $PerlIO::encoding::fallback when the layer is applied, so you can set that to one of the Encoding::FB_* values.

[xdg]

On reflection, I'm going to close this "won't fix". Users can disable warnings in various ways if as a Tiny module, I don't think it's the right move to add callback overhead handling malformed characters.