Comments (15)
Hmmm, UNOB does not suppose to use UTF-8 characters, right?
That is the reason, parser wants to strip as defined by the standard, so, technically your parser does what is right, but the file itself is UNOY.
So, my example file has wrong UNB header content...
from edifact.
Hey all.
If somebody need UNOE for Cyrillic:
$parser->setStripRegex("/[\x20-\x7E]\xA0-\xFF/");
from edifact.
My EDI file starts with this:
UNA:+.? '
UNB+UNOB:2+TECHNIQ+TERMIQUE+180720:1105+957'
from edifact.
I succeed with a brute force workaround:
I commented out the following line to do not remove any characters:
Line 139 in 6fc2fe5
from edifact.
Could you send the file to my email, so I can try? :)
from edifact.
Ok, the producer of my input file fixed the UNB to include the UNOY instead of UNOB and fixed some other smaller issues.
So, as a summary:
- using utf8 encoded messages are possible, but must check the UNB segment to have UNOY
- it seems, hacking the code a little bit can force the utf8 support, but must check the result
Question:
As I can see, Basic sanitization, remove non printable chars always will process the input line, even UNOY will be set and legitimate utf-8 chars exist.
Here you can see the code, which will run before UNB (encoding processed):
Lines 136 to 139 in 6fc2fe5
What if stripchars will be set after UNB line processed and apply stripchars depending on the encodings?
from edifact.
Yeah, I don't test actually files with different encodings, thanks for the file, whenever I can I look at it more.
The stripChars variable theoretically could be set before parsing with setStripRegex(), but as I was checking on your file I didn't find a regex that works (tried /[\x01-\x09\x0B-\x0C\x0E-\x1F\x7F-\x9F]/).
Surely needs changing, if you find a nice solution before me it's welcome :-)
from edifact.
Depending on the UNB settings in the file, I think one scenario should work:
- new Reader/Parser without actual file or string
- setStripRegex() --- force ignore UNB settings
- parser->loadString() --- if stripregex is empty, practically preg_replace will not harm the string
At this moment it seems the basic sanitization always will happen. Maybe at the weekend I will have time to play with it.
from edifact.
This isn't actually fixed, is it? Please keep this open, if it isn't. If only to warn others of expected problems.
I'm trying to parse a file (Header starts with UNB+UNOC:3
) with German characters like ß
in them, and
Line 191 in c577183
replaces them with non-printable characters and then complains about it. Characters like äöü
already arrive broken in the method. The funny part is that those ORDERS files were generated by the php-edifact/edifact-generator with those letters seemingly intact.
What exactly is missing for this to work as expected?
I've found this on the web: https://blog.sandro-pereira.com/2009/08/15/edifact-encoding-edi-character-set-support/ and according to https://en.wikipedia.org/wiki/ISO/IEC_8859 UNOC
should allow for those characters to exist because it refers to ISO-8859-1 (Latin-1) encoding.
https://groups.google.com/g/botsmail/c/B6V5mwdcFts/m/DRcsE_K7BgAJ claims that UNOW is for UTF-8 while UNOY seems to be the whole UTF-32, or something… But apparently they only exist as part of syntax version 4, for which I've found this: https://www.gefeg.com/jswg/v4/data/v4_docs.htm
from edifact.
Ok, I reopen it... I never had some file to actually test, if you can send me one via email I can look...
Perhaps we should use multibythe versions of the various functions?
from edifact.
Do your classes support syntax v4? If so, one could simply set UNOW/Y, pass UTF-8 encoded strings and skip the char replacement. Should theoretically come out fine.
But fixing it so UNOC v3 actually allows the supported characters to come out correctly makes sense, too.
Right now it seems like your EDI classes don't concern themselves with text encoding whatsoever, do they?
from edifact.
Actualy I don't know, I usually process EDI v3 only so I never looked in detail... Future work :)
For encoding, I only made sure to strip out invalid chars according to the required set (although for example I don't bother about uppercase and one supplier one time complained I sent lowercase characters haha)
Someone tried to use utf8_encode if I recall, although that function is deprecated in php 8.1
from edifact.
I have to partially retract my statements. It works fine, if you actually feed it text in one of the explicitly supported character encodings. I was feeding it UTF-8. 🤦♂️ My fault for not knowing enough about EDI. Sorry.
Now I think there should be a warning somewhere that UTF-8 isn't supported. Maybe a trigger_error()
in the Parser::loadString()
method? Something like:
// Unicode is only supported starting with UNOW syntax which requires syntax v4
// neither are currently (fully?) supported
if (mb_check_encoding($string, 'UTF-8') === true)
trigger_error('UTF-8 encoded text found', E_USER_WARNING);
After all, PHP itself usually assumes UTF-8 encoding everywhere, if you don't specify something else. But I think any kind of text conversion should be left to the user.
from edifact.
I would like to only populate the errors array, user should break execution explicitly by checking that... Is there some way of silently detecting and converting the file?
I would like to test it if you can share a sample...
Currently Parser detects some metadata on the file (although potentially that won't properly work when there's an interchange with multiple messages)
I can look at it in the next days probably
from edifact.
A UTF-8 file might still come out fine as long as it contains only letters within the first 127 code points, like in English for example. So breaking execution in that case is overly strict, that's why I chose a mere warning instead.
Like I said, I'm not a fan of doing things silently in the background. Detecting character encoding with 100% certainty is impossible anyway. Only the creator of the text can know its encoding for sure. So better not mess with it and use it as is instead.
You could only check, if a text matches a certain encoding. So you'd have to write code in the sense of "if the file says it's UNOC, check if it passes an ISO-8859-1 encoding check" and so on.
from edifact.
Related Issues (20)
- Release 0.5.0 not available via Composer HOT 2
- Possibility to have IDs instead of name as output from Interpreter::prepare HOT 5
- CDR read HOT 4
- Problems reading RFF segment from EDI-File HOT 1
- Array to string conversion, if an epty array met. HOT 1
- X12 HOT 6
- Package ramsey/array_column is abandoned HOT 2
- Trouble understanding how to get the data we need HOT 5
- Segment is ambiguous HOT 1
- Elements with maxrepeat > 1 should be considered arrays when parsed as json HOT 6
- Reader doesn't return full multi-line texts HOT 4
- Multiple SG2 segments not allowed HOT 8
- Does IFTMIN:S:93A supported? HOT 5
- How to create edi document with .edi extension for healthcare HOT 4
- readEdiSegmentDTM for SG12 HOT 5
- File has multiple messages HOT 6
- question marks are not parsed correctly HOT 5
- Fatal error on integer-value segments for Encoder HOT 2
- Reader cannot be init with a null param HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from edifact.