Comments (7)
FEFF
is the big-endian UTF-16 BOM. Seems reasonable to only support UTF-8 encoded files to me.
from cstparser.jl.
Same thing seems to show up many places.
https://github.com/SciML/OrdinaryDiffEq.jl/runs/5853304680?check_suite_focus=true#step:4:104
https://github.com/SciML/OrdinaryDiffEq.jl/blob/v6.8.0/src/OrdinaryDiffEq.jl#L303
etc.
from cstparser.jl.
The culprit here seems to be a format character at the start of the file, which Julia's parser just ignores:
julia> CSTParser.parse("\ufeffusing Test")
1:13 errortoken
1:3 errortoken( CSTParser.UnexpectedToken)
1:3 errortoken( CSTParser.Unknown)
4:13 using
4:7 1:0 OP: .
4:7 Test
julia> Meta.parse("\ufeffusing Test")
:(using Test)
from cstparser.jl.
Nvm, this actually is UTF-8 with BOM:
julia> first(codeunits(read("src/OrdinaryDiffEq.jl", String)), 6)
6-element Vector{UInt8}:
0xef
0xbb
0xbf
0x22
0x22
0x22
It's somewhat arguable that we should support that.
from cstparser.jl.
Julia itself just treats U+FEFF as a space:
julia> Meta.parse("[1\ufeff2]")
:([1 2])
That's a simple approach that allows it as a BOM as well since parsing doesn't care about space at the beginning of a file.
from cstparser.jl.
JuliaLang/Tokenize.jl#197 does that.
One potential issue with that approach is that JuliaFormatter is likely to remove the unnecessary leading space, which some Microsoft tools apparently don't like.
from cstparser.jl.
Maybe it would make sense to treat a leading BOM as a special token that JuliaFormatter knows to retain. Or the logic could be changed from deleting leading whitespace to trimming leading whitespace to just the BOM if there is a BOM.
from cstparser.jl.
Related Issues (20)
- Parsing `@doc` macro with trailing newline errors HOT 1
- Stop parsing when it occurs to lambda function assignment HOT 2
- Return information about when state errored HOT 2
- `const` field support
- Incorrect Parse in global statement HOT 1
- Possible error in import parsing HOT 1
- Parser error with raw string with $ followed by multiline string
- Incorrect parsing with `global const ...`
- CSTParser cannot parse for loops in a quoted macro
- Command parsed possibly incorrectly HOT 3
- Multidimensional parse issue HOT 2
- crash on global 'tuple' in let block HOT 2
- language server crashing within lint! given hcat input HOT 3
- Problem parsing in JuliaFormatter HOT 3
- Parsing error on `$a'` HOT 1
- Parsing list comprehensions
- Failure to parse string macros with module name & suffix (`X.y"z"q`)
- ERROR: BoundsError: attempt to access 1-element Vector{CSTParser.EXPR} at index [2] HOT 4
- crashes on nightly Julia, stateful iterators don't have `length` any more HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cstparser.jl.