PHP Numeric literals

[scintillaorg]

PHP Numeric literals

[ivan-u7n]

It turned out that my earlier implementation was over zealous. This one seems to be less so. It still consumes as many word characters as it can, but its eagerness for operator characters is now strictly regulated.

The test

123456; 123_456; // +
1234z6; 123456_; 123__456; // -

0x89Ab; 0x89_aB; // +
0x89zB; 0x89AB_; 0x_89AB; 0_x89AB; 0x89__AB; // -

1234.; 1234.e-0; 1234e+0; 1234e0; // +
1234.e-; 1234e+; 1234e; // -

.1234; .12e0; // +
.12.0e0; .12e0.0; .12e0e0; // -

1.234e-10; 1.2_34e-1_0; // +
1.234e-_10; 1.234e_-10; 1.234_e-10; 1._234e-10; 1_.234e-10; // -

01234567; 0_1234567; // +
012345678; // -

0...0; // 0. . .0
.0..0; // .0 . .0
0e+0+0e+0; // 0e+0 + 0e+0

[nyamatongwe]

Due to reevaluating octal numbers, i is reset and may count a following '/*' comment double for folding. This caused the apostophe.php test case to fail with this change, the fold level jumping from 401 to 403:

 2 400   0 + <?php
 0 401   0 | # Test that currently fails as comment style not started in a number.
 0 401   0 | # line-comment
 0 401   0 | // line-comment
 0 401   0 | /* comment */
 0 401   0 | $foo = 0#comment
 0 401   0 | $foo = 0//comment
 2 401   0 + $foo = 0/*'*/;
 0 403   0 | ?>
 1 403   0 | 
 0 403   0 | <br />
 0 403   0 | 

Rewinding is often a difficult strategy as there are multiple effects occurring in the code.

[nyamatongwe]

I agree that the "HTML" lexer is rather complicated and needs overhaul. However, do you have in mind compile-time or run-time combining?

Sets of choices would be defined at compile time. At runtime an application may say they want to combine ASP and C# for server-side scripting along with Mustache and Sass then Dart client side. A limited number of slots (server-side-control, server-side-scripting; expression-language; stylesheet-language; client-side-language) would be specified at compile-time along with the choices available in each slot. The choice for each slot would be set by the application at runtime.

[ivan-u7n]

Due to reevaluating octal numbers, i is reset and may count a following '/*' comment double for folding.

The revised patch fixes it by using the inline reevaluation.

Rewinding is often a difficult strategy as there are multiple effects occurring in the code.

I forgot that there are side-effects in the lex loop.

The patch also fixes the apostophe.php test case entirely by reusing the code in PHP's default branch, although it does so by using goto, which isn't liked by everybody. But this case seems perfect for this often misused language feature.

Here is how it might look like:

[nyamatongwe]

Issues and their fixes should be isolated. If the current apostophe.php case is incorrect then that should be a separate issue and fix.

LexerHTML::Lex is 1500 lines long and is worked on by many people so is a bad place to use a goto.

It should be possible to move the character handling logic for numeric literals out of Lex and into PhpNumberState.

[ivan-u7n]

Here is the next iteration of my patch.

All character handling logic is moved into PhpNumberState. I also removed the usage of goto.

However the failing tests in test/examples/hypertext/apostophe.php still got fixed as a byproduct of using the look-ahead strategy in PhpNumberState.

The test case

123456; 123_456; // +
1234z6; 123456_; 123__456; // -

0x89Ab; 0x89_aB; // +
0x89zB; 0x89AB_; 0x_89AB; 0_x89AB; 0x89__AB; // -

1234.; 1234.e-0; 1234e+0; 1234e0; 1234e0.PHP_EOL; // +
1234._; 1234.e-; 1234e+; 1234e; // -

.1234; .12e0; // +
.12.0e0; .12e0.0; .12e0e0; // -

1.234e-10; 1.2_34e-1_0; // +
1.234e-_10; 1.234e_-10; 1.234_e-10; 1._234e-10; 1_.234e-10; // -
1.234e-+10; 1.234e-_+10; // -

01234567; 0_1234567; // +
012345678; // -

0...0; // 0. . .0
.0..0; // .0 . .0
0e+0+0e+0; // 0e+0 + 0e+0

;0#comment
;0//comment
;0/*comment*/;

should render like

PS. I think, I'll open another issue for the hypertext-lexer-overhaul discussion after I think it through, for I have some ideas but I'm not sure whether they are implementable.

[nyamatongwe]

Committed with reformatting. LexHTML.cxx follows 1TBS as does most of Lexilla and Scintilla. Changes should match existing code. Also fixed spelling of "unchanged".