Fix matching of `\N`

[apple]

Fix matching of `\N`

[hamishknight]

cc @natecook1000, I also noticed \N doesn't appear to be mentioned in the Unicode proposal, should it be?

[natecook1000]

Oh, interesting — I didn't know that behavior existed! I'll get it added to the Unicode proposal, yes.

[natecook1000]

Actually, do you have an example of how this isn't working how you'd expect? If \N is supposed to be an inverse of \n, which I think we're going to treat basically like \R, then it should also be affected by things like .asciiOnlyWhitespace().

[hamishknight]

Hmm, it was my understanding that it followed .'s newline behavior (i.e not being affected by ASCII-only whitespace). PCRE defines it as:

The escape sequence \N when not followed by an opening brace behaves like a dot, except that it is not affected by the PCRE2_DOTALL option. In other words, it matches any character except one that signifies the end of a line.

[hamishknight]

Oniguruma defines it as:

\N       negative newline  (?-m:.)

which is now extra confusing because Oniguruma seems to define (?m) differently to PCRE?

 m: multi-line (dot (.) also matches newline)

[hamishknight]

That being said, I'm fine if we want to define it as the inverse of \R. It should however be noted that PCRE supports changing the definition of \R through global matching options such as (*BSR_ANYCRLF) (which wouldn't affect \N in PCRE AFAIK). I'm not sure if we want to support those global matching options tho.

[natecook1000]

I think this is the tricky bit:

except one that signifies the end of a line

In addition to changing the behavior of . and \s, does switching option modes also change what characters signify the end of a line? I'd expect that "the end of a line" would be equivalent to what \R matches. If you only want to recognize ASCII whitespace, you don't want to let non-ASCII line-break character define where lines start and end.

[hamishknight]

PCRE at least defines newline conventions and \R differently. Newline conventions are affected by these global options (under the heading "Newline conventions"):

  (*CR)        carriage return
  (*LF)        linefeed
  (*CRLF)      carriage return, followed by linefeed
  (*ANYCRLF)   any of the three above
  (*ANY)       all Unicode newline sequences
  (*NUL)       the NUL character (binary zero)

And PCRE says:

The newline convention affects where the circumflex and dollar assertions are true. It also affects the interpretation of the dot metacharacter when PCRE2_DOTALL is not set, and the behaviour of \N when not followed by an opening brace. However, it does not affect what the \R escape sequence matches. By default, this is any Unicode newline sequence, for Perl compatibility. However, this can be changed; see the next section and the description of \R in the section entitled "Newline sequences" below. A change of \R setting can be combined with a change of newline convention.