sheetjs/js-codepage

:currency_exchange: Codepages for JS

js-codepage

Codepages are character encodings. In many contexts, single- or double-byte character sets are used in lieu of Unicode encodings. The codepages map between characters and numbers.

Setup

In node:

var cptable = require('codepage');

In the browser:

<script src="cptable.js"></script>
<script src="cputils.js"></script>

Alternatively, use the full version in the dist folder:

<script src="cptable.full.js"></script>

The complete set of codepages is large due to some Double Byte Character Set encodings. A much smaller file that only includes SBCS codepages is provided in this repo (sbcs.js), as well as a file for other projects (cpexcel.js)

If you know which codepages you need, you can include individual scripts for each codepage. The individual files are provided in the bits/ directory. For example, to include only the Mac codepages:

<script src="bits/10000.js"></script>
<script src="bits/10006.js"></script>
<script src="bits/10007.js"></script>
<script src="bits/10029.js"></script>
<script src="bits/10079.js"></script>
<script src="bits/10081.js"></script>

All of the browser scripts define and append to the cptable object. To rename the object, edit the JSVAR shell variable in make.sh and run the script.

The utilities functions are contained in cputils.js, which assumes that the appropriate codepage scripts were loaded.

The script will manipulate module.exports if available . This is not always desirable. To prevent the behavior, define DO_NOT_EXPORT_CODEPAGE.

Usage

Most codepages are indexed by number. To get the Unicode character for a given codepoint, use the dec property:

var unicode_cp10000_255 = cptable[10000].dec[255]; // ˇ

To get the codepoint for a given character, use the enc property:

var cp10000_711 = cptable[10000].enc[String.fromCharCode(711)]; // 255

There are a few utilities that deal with strings and buffers:

var 汇总 = cptable.utils.decode(936, [0xbb,0xe3,0xd7,0xdc]);
var buf =  cptable.utils.encode(936,  汇总);
var sushi= cptable.utils.decode(65001, [0xf0,0x9f,0x8d,0xa3]); // 🍣
var sbuf = cptable.utils.encode(65001, sushi);

cptable.utils.encode(CP, data, ofmt) accepts a String or Array of characters and returns a representation controlled by ofmt:

• Default output is a Buffer (or Array) of bytes (integers between 0 and 255)
• If ofmt == 'str', return a binary String (byte i is o.charCodeAt(i))
• If ofmt == 'arr', return an Array of bytes

cptable.utils.decode(CP, data) accepts a byte String or Array of numbers or Buffer and returns a JS string.

Known Excel Codepages

A much smaller script, including only the codepages known to be used in Excel, is available under the name cpexcel. It exposes the same variable cptable and is suitable as a drop-in replacement when the full codepage tables are not needed.

In node:

var cptable = require('codepage/dist/cpexcel.full');

Rolling your own script

The make.sh script in the repo can take a manifest and generate JS source.

Usage:

$ bash make.sh path_to_manifest output_file_name JSVAR

where

• JSVAR is the name of the exported variable (generally cptable)
• output_file_name is the output file (cpexcel.js, cptable.js, ...)
• path_to_manifest is the path to the manifest file.

The manifest file is expected to be a CSV with 3 columns:

<codepage number>,<source>,<size>

If a source is specified, it will try to download the specified file and parse. The file format is expected to follow the format from the unicode.org site. The size should be 1 for a single-byte codepage and 2 for a double-byte codepage. For mixed codepages (which use some single- and some double-byte codes), the script assumes the mapping is a prefix code and generates efficient JS code.

Generated scripts only include the mapping. cat a mapping with cputils.js to produce a complete script like cpexcel.full.js.

Building the complete script

This script uses voc. The script to build the codepage tables and the JS source is codepage.md, so building involves voc codepage.md.

Generated Codepages

The complete list of codepages can be found in the file pages.csv.

Some codepages are easier to implement algorithmically. Since those character tables are not generated, there is no corresponding entry (they are "magic").

CP#	Source	Description
37	unicode.org	IBM EBCDIC US-Canada
437	unicode.org	OEM United States
500	unicode.org	IBM EBCDIC International
620	NLS	Mazovia (Polish) MS-DOS
708	Windows 7	Arabic (ASMO 708)
720	Windows 7	Arabic (Transparent ASMO); Arabic (DOS)
737	unicode.org	OEM Greek (formerly 437G); Greek (DOS)
775	unicode.org	OEM Baltic; Baltic (DOS)
808	unicode.org	OEM Russian; Cyrillic + Euro symbol
850	unicode.org	OEM Multilingual Latin 1; Western European (DOS)
852	unicode.org	OEM Latin 2; Central European (DOS)
855	unicode.org	OEM Cyrillic (primarily Russian)
857	unicode.org	OEM Turkish; Turkish (DOS)
858	Windows 7	OEM Multilingual Latin 1 + Euro symbol
860	unicode.org	OEM Portuguese; Portuguese (DOS)
861	unicode.org	OEM Icelandic; Icelandic (DOS)
862	unicode.org	OEM Hebrew; Hebrew (DOS)
863	unicode.org	OEM French Canadian; French Canadian (DOS)
864	unicode.org	OEM Arabic; Arabic (864)
865	unicode.org	OEM Nordic; Nordic (DOS)
866	unicode.org	OEM Russian; Cyrillic (DOS)
869	unicode.org	OEM Modern Greek; Greek, Modern (DOS)
870	Windows 7	IBM EBCDIC Multilingual/ROECE (Latin 2)
872	unicode.org	OEM Cyrillic (primarily Russian) + Euro Symbol
874	unicode.org	Windows Thai
875	unicode.org	IBM EBCDIC Greek Modern
895	NLS	Kamenický (Czech) MS-DOS
932	unicode.org	Japanese Shift-JIS
936	unicode.org	Simplified Chinese GBK
949	unicode.org	Korean
950	unicode.org	Traditional Chinese Big5
1010	IBM	IBM EBCDIC French
1026	unicode.org	IBM EBCDIC Turkish (Latin 5)
1047	Windows 7	IBM EBCDIC Latin 1/Open System
1132	IBM	IBM EBCDIC Lao (1132 / 1133 / 1341)
1140	Windows 7	IBM EBCDIC US-Canada (037 + Euro symbol)
1141	Windows 7	IBM EBCDIC Germany (20273 + Euro symbol)
1142	Windows 7	IBM EBCDIC Denmark-Norway (20277 + Euro symbol)
1143	Windows 7	IBM EBCDIC Finland-Sweden (20278 + Euro symbol)
1144	Windows 7	IBM EBCDIC Italy (20280 + Euro symbol)
1145	Windows 7	IBM EBCDIC Latin America-Spain (20284 + Euro symbol)
1146	Windows 7	IBM EBCDIC United Kingdom (20285 + Euro symbol)
1147	Windows 7	IBM EBCDIC France (20297 + Euro symbol)
1148	Windows 7	IBM EBCDIC International (500 + Euro symbol)
1149	Windows 7	IBM EBCDIC Icelandic (20871 + Euro symbol)
1200	magic	Unicode UTF-16, little endian (BMP of ISO 10646)
1201	magic	Unicode UTF-16, big endian
1250	unicode.org	Windows Central Europe
1251	unicode.org	Windows Cyrillic
1252	unicode.org	Windows Latin I
1253	unicode.org	Windows Greek
1254	unicode.org	Windows Turkish
1255	unicode.org	Windows Hebrew
1256	unicode.org	Windows Arabic
1257	unicode.org	Windows Baltic
1258	unicode.org	Windows Vietnam
1361	Windows 7	Korean (Johab)
10000	unicode.org	MAC Roman
10001	Windows 7	Japanese (Mac)
10002	Windows 7	MAC Traditional Chinese (Big5)
10003	Windows 7	Korean (Mac)
10004	Windows 7	Arabic (Mac)
10005	Windows 7	Hebrew (Mac)
10006	unicode.org	Greek (Mac)
10007	unicode.org	Cyrillic (Mac)
10008	Windows 7	MAC Simplified Chinese (GB 2312)
10010	Windows 7	Romanian (Mac)
10017	Windows 7	Ukrainian (Mac)
10021	Windows 7	Thai (Mac)
10029	unicode.org	MAC Latin 2 (Central European)
10079	unicode.org	Icelandic (Mac)
10081	unicode.org	Turkish (Mac)
10082	Windows 7	Croatian (Mac)
12000	magic	Unicode UTF-32, little endian byte order
12001	magic	Unicode UTF-32, big endian byte order
20000	Windows 7	CNS Taiwan (Chinese Traditional)
20001	Windows 7	TCA Taiwan
20002	Windows 7	ETEN Taiwan (Chinese Traditional)
20003	Windows 7	IBM5550 Taiwan
20004	Windows 7	TeleText Taiwan
20005	Windows 7	Wang Taiwan
20105	Windows 7	Western European IA5 (IRV International Alphabet 5)
20106	Windows 7	IA5 German (7-bit)
20107	Windows 7	IA5 Swedish (7-bit)
20108	Windows 7	IA5 Norwegian (7-bit)
20127	magic	US-ASCII (7-bit)
20261	Windows 7	T.61
20269	Windows 7	ISO 6937 Non-Spacing Accent
20273	Windows 7	IBM EBCDIC Germany
20277	Windows 7	IBM EBCDIC Denmark-Norway
20278	Windows 7	IBM EBCDIC Finland-Sweden
20280	Windows 7	IBM EBCDIC Italy
20284	Windows 7	IBM EBCDIC Latin America-Spain
20285	Windows 7	IBM EBCDIC United Kingdom
20290	Windows 7	IBM EBCDIC Japanese Katakana Extended
20297	Windows 7	IBM EBCDIC France
20420	Windows 7	IBM EBCDIC Arabic
20423	Windows 7	IBM EBCDIC Greek
20424	Windows 7	IBM EBCDIC Hebrew
20833	Windows 7	IBM EBCDIC Korean Extended
20838	Windows 7	IBM EBCDIC Thai
20866	Windows 7	Russian Cyrillic (KOI8-R)
20871	Windows 7	IBM EBCDIC Icelandic
20880	Windows 7	IBM EBCDIC Cyrillic Russian
20905	Windows 7	IBM EBCDIC Turkish
20924	Windows 7	IBM EBCDIC Latin 1/Open System (1047 + Euro symbol)
20932	Windows 7	Japanese (JIS 0208-1990 and 0212-1990)
20936	Windows 7	Simplified Chinese (GB2312-80)
20949	Windows 7	Korean Wansung
21025	Windows 7	IBM EBCDIC Cyrillic Serbian-Bulgarian
21027	NLS	Extended/Ext Alpha Lowercase
21866	Windows 7	Ukrainian Cyrillic (KOI8-U)
28591	unicode.org	ISO 8859-1 Latin 1 (Western European)
28592	unicode.org	ISO 8859-2 Latin 2 (Central European)
28593	unicode.org	ISO 8859-3 Latin 3
28594	unicode.org	ISO 8859-4 Baltic
28595	unicode.org	ISO 8859-5 Cyrillic
28596	unicode.org	ISO 8859-6 Arabic
28597	unicode.org	ISO 8859-7 Greek
28598	unicode.org	ISO 8859-8 Hebrew (ISO-Visual)
28599	unicode.org	ISO 8859-9 Turkish
28600	unicode.org	ISO 8859-10 Latin 6
28601	unicode.org	ISO 8859-11 Latin (Thai)
28603	unicode.org	ISO 8859-13 Latin 7 (Estonian)
28604	unicode.org	ISO 8859-14 Latin 8 (Celtic)
28605	unicode.org	ISO 8859-15 Latin 9
28606	unicode.org	ISO 8859-15 Latin 10
29001	Windows 7	Europa 3
38598	Windows 7	ISO 8859-8 Hebrew (ISO-Logical)
47451	unicode.org	Atari ST/TT
50220	magic	ISO 2022 JIS Japanese with no halfwidth Katakana
50221	magic	ISO 2022 JIS Japanese with halfwidth Katakana
50222	magic	ISO 2022 Japanese JIS X 0201-1989 (1 byte Kana-SO/SI)
50225	magic	ISO 2022 Korean
50227	magic	ISO 2022 Simplified Chinese
51932	Windows 7	EUC Japanese
51936	Windows 7	EUC Simplified Chinese
51949	Windows 7	EUC Korean
52936	Windows 7	HZ-GB2312 Simplified Chinese
54936	Windows 7	GB18030 Simplified Chinese (4 byte)
57002	Windows 7	ISCII Devanagari
57003	Windows 7	ISCII Bengali
57004	Windows 7	ISCII Tamil
57005	Windows 7	ISCII Telugu
57006	Windows 7	ISCII Assamese
57007	Windows 7	ISCII Oriya
57008	Windows 7	ISCII Kannada
57009	Windows 7	ISCII Malayalam
57010	Windows 7	ISCII Gujarati
57011	Windows 7	ISCII Punjabi
65000	magic	Unicode (UTF-7)
65001	magic	Unicode (UTF-8)

unicode.org refers to the Unicode Consortium Public Mappings, a database of various mappings between Unicode characters and respective character sets. The tables are processed by a few scripts in the build process.

IBM refers to the IBM coded character set database. Even though IBM uses a different numbering scheme from Windows, the IBM numbers are used when there is no conflict. The tables are manually generated from the symbol manifests.

Windows 7 refers to direct inspection of Windows 7 machines using .NET class System.Text.Encoding. The enclosed MakeEncoding.cs C# program brute-forces code pages. MakeEncoding.cs deviates from unicode.org in some cases. When they map a given code to different characters, unicode.org value is used. When unicode.org does not prescribe a value, MakeEncoding.cs value is used.

NLS refers to the National Language Support files supplied in various versions of Windows. In older versions of Windows (like Windows 98) these files followed the name pattern CP_#.NLS, but newer versions use the name pattern C_#.NLS.

Testing

make test will run the nodejs-based test.

To run the in-browser tests, run a local server and go to the ctest directory. make ctestserv will start a python SimpleHTTPServer server on port 8000.

To update the browser artifacts, run make ctest.

Sources

• Unicode Consortium Public Mappings
• Windows Code Page Enumeration
• Windows Code Page Identifiers
• IBM Coded Character Sets
• ISO/IEC 2022 / ECMA-35
• International Register of Coded Character Sets To Be Used With Escape Sequences
• Japanese Character Encoding for Internet Messages

License

Please consult the attached LICENSE file for details. All rights not explicitly granted by the Apache 2.0 license are reserved by the Original Author.

Badges

js-codepage's Issues

The file is huge

Is there a way to reduce the disk footprint?
For example, adding separate scripts for encoding/decoding, using raw characters instead of number strings, using a minifier, etc.

As a side note, it might also be useful to introduce an efficient decoding function, e.g.:

cp.decode('ÇÈÉÊ', 1251)

It might be even possible to just use arrays. Index would be the character code. Although it would result in a waste of space for the first 127 characters, so it's comparable to using objects.

Include sbcs.full.js in Node Package

I've found I can get by with sbcs.full.js and not need to load the 2mb cptable.js. However sbcs.full.js isn't available via npm (that I can find). Can we get it added to the package?

Encoding problem with Japanese

Hi,

Here is the result when I try to import this xls file
(with japanese caracter) :

I have the same result when I try it in my application.

Is there a way to have an UTF-8 encoding ?

farsi(persian) codepage for SheetJS

hi
I'm trying to convert an excel file(input type file) to html table. my code does work but it shows most of the letters in unreadable format because most of my excel cells are in farsi and farsi letters are special characters. I know that some arabic codepages cover farsi letters (such as 1256) but my code doesn't show letters correctly even after i use codepage
please help

<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <script src="jquery.min.js"></script>
    <script lang="javascript" src="xlsx.full.min.js"></script>
    <script src="JavaScript.js" type="text/javascript"></script>
    <script src="cpexcel.js"></script>
    <script src="cptable.js"></script>
    <script src="1256.js"></script>
    <meta charset="UTF-8" />
</head>
<body>
    <div id="navbar"><span>Red Stapler - SheetJS </span></div>
<div id="wrapper">
        <input type="file" id="input-excel" />
</div>
<script>
        $('#input-excel').change(function(e){
                var reader = new FileReader();
                reader.readAsArrayBuffer(e.target.files[0]);
                reader.onload = function(e) {
                    var data = new Uint8Array(reader.result);
                    var wb = XLSX.read(data, { type: 'array', codepage: 1256,charset:'UTF-8'});
                    var htmlstr = XLSX.write(wb, { sheetIndex:1 , type: 'binary', bookType: 'html',});
                        $('#wrapper')[0].innerHTML += htmlstr;
                }
        });
</script>
</body>
</html>

ISO 2022 JIS Japanese encoding fails

Hi, thanks very much for your work on this repository, it's incredibly useful. We use it as the main character encoding library for CyberChef.

We've recently noticed an issue when trying to encode into ISO 2022 JIS Japanese where only null bytes are returned.

The affected CP numbers are 50220, 50221 and 50222.

Example code

import cptable from "codepage";

cptable.utils.encode(50220, "こんにちは");

Expected output

Uint8Array(10) [164, 179, 164, 243, 164, 203, 164, 193, 164, 207]

Actual output

Uint8Array(5) [0, 0, 0, 0, 0]

Can you shed any light on this behaviour?

cannot use this module on Typescript

export defualt X is exports['default'] = X
export defualt X is not module.exports = X
import X from 'x' is const X = require('x')['default']
import X from 'x' is not const X = require('x')

#19

ensure scripts play nice with browserify

See SheetJS/sheetjs#76

Add bits scripts to module

Right now I'm using sbcs due to its small size. I would like the flexibility to add a few missing scripts via the bits files instead of flipping to the full table. Could we get the bits dir added back to the module to aid in this?

Perhaps typescript would be better option? Would be interested to have a smaller bundle.

Perhaps typescript would be better option? Would be interested to have a smaller bundle.

We can get more mileage if we focus on the cpexcel as its the largest part.

Noticed the cpexcel.js can probably be reduced in size as there's many repeated strings like we can have the common prefix as separate variable.

Below constant is repeated 31 times, which can reduce about 8K characters.

\u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\b\t\n\u000b\f\r\u000e\u000f\u0010\u0011\u0012\u0013\u0014\u0015\u0016\u0017\u0018\u0019\u001a\u001b\u001c\u001d\u001e\u001f !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~

The below scripts can be created as factory / common functions

Below is called 26 times - can reduce about 4K

, D = [], e = {}; for(var i=0;i!=d.length;++i) { if(d.charCodeAt(i) !== 0xFFFD) e[d.charAt(i)] = i; D[i] = d.charAt(i); } return {"enc": e, "dec": D }; })();

Below have 468 calls to it - about 50K reduction

for(j = 0; j != D[185].length; ++j) if(D[185][j].charCodeAt(0) !== 0xFFFD) { e[D[185][j]] = 47360 + j; d[47360 + j] = D[185][j];}

The above 3 combined, we can easily shave off 60K+ characters from the script.

I think the above should be still maintaining the readability of the current code.

Originally posted by @beeing in SheetJS/sheetjs#2085 (comment)

NodeJs v.6.0.0 introduces issue with cpexcel.js Buffer usage.

Setting the length property after construction of Buffer objects is no longer valid.

Reading an excel spreadsheet buffer yields:

TypeError: Cannot set property length of [object Object] which has only a getter
at sbcs_d (.../node/node_modules/xlsx/dist/cpexcel.js:909:20)
at Object.decode (.../node_modules/xlsx/dist/cpexcel.js:1197:40)
at _gc2 (.../node/node_modules/xlsx/xlsx.js:28:24)
at Buffer.ReadShift as read_shift
at parse_XLUnicodeStringNoCch (.../node/node_modules/xlsx/xlsx.js:3113:23)
at parse_XLUnicodeString (.../node/node_modules/xlsx/xlsx.js:3121:9)
at parse_XLUnicodeString2 (.../node/node_modules/xlsx/xlsx.js:3125:48)
at Object.parse_Format as f
at slurp (.../node/node_modules/xlsx/xlsx.js:9229:11)
at parse_workbook (.../node/node_modules/xlsx/xlsx.js:9350:15) +0ms
.../node/node_modules/xlsx/xlsx.js:9350:15)

Just removing the statement at line 909 seems to make everything work again, but not sure how safe this is without fully understanding the context. Hoping someone with more knowledge of this library can provide the right fix.

Unicode characters and codepages

Implement code pages properly (should be a separate MIT-licensed project)

Properly emit unicode characters (and I think that xls2csv should be returning unicode as well)

Add option to write BOM for unicode types

See discussion at SheetJS/sheetjs#78

weird chinese text in production deployment

Just an example: 情悻悵惜悼惘惕惆惟悸惚惇戚戛扈掠控捲掖探 but there are tens if not hundrets.
Found it in my production code, as I use amcharts who use xlsx who use codepage. You may want to check that garbage text and remove it from your code.

It's about encoding issue.

I'm a Korean developer using Sheetsjs.
I'd like to export file in csv. For that, I used sheet_to_csv util.
And I need to have the csv file encoded in euc-kr, now it's utf-8 with BOM.
And I also know that codepage utils. But I couldn't find any euc-kr type in codepage utils.
So I tried to do that with cp949. But I failed to encode hangul.
Here's my code and screen shot.

How can I solve this issue?

var encData = cptable.utils.encode(949, sheet, 'str'); console.log(encData);

and its results are below.

Thank you for your cooperation in advance.

README.md typo

at codepage 1253 Green should be Greek

43MB is kind of big

is there any way to reduce the size of this module? e.g. npm install j ends up being a 100mb after npm install because it installs codepage twice

Error with special characters in Codepage 850 (and others?)

Hi,
thanks for the package, using it to drive an LCD display from a JS. The LCD is using Codepage 850 and basically text and numbers are working fine.
However, when I try to send special characters, e.g. the smiley face "☺︎" (u+263A), it does not work.
So I looked into the code of the js-codepage package to see why:

I think, there is an error in the lookup string that is used to for the encode function.
In line 9 of cptable.js there is this definition:

cptable[850] = (function(){ var d = "\u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\b\t\n\u000b\f\r\u000e\u000f\u0010\u0011\u0012\u0013\u0014\u0015\u0016\u0017\u0018\u0019\u001a\u001b\u001c\u001d\u001e\u001f !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~�ÇüéâäàåçêëèïîìÄÅÉæÆôöòûùÿÖÜø£Ø׃áíóúñѪº¿®¬½¼¡«»░▒▓│┤ÁÂÀ©╣║╗╝¢¥┐└┴┬├─┼ãÃ╚╔╩╦╠═╬¤ðÐÊËÈıÍÎÏ┘┌█▄¦Ì▀ÓßÔÒõÕµþÞÚÛÙýݯ´­±‗¾¶§÷¸°¨·¹³²■ ", D = [], e = {}; for(var i=0;i!=d.length;++i) { if(d.charCodeAt(i) !== 0xFFFD) e[d.charAt(i)] = i; D[i] = d.charAt(i); } return {"enc": e, "dec": D }; })();

However, the Unicode characters at the beginning of the script are not correct.
The second character in Codepage 850 is not Unicode u+0001 but Unicode u+263A, the next should be u+263B instead of u+0002.
So line 9 should start like:

cptable[850] = (function(){ var d = "\u0000\u263A\u263B\u2665\u2666\u2663\u2660\u2022 [...]

I verified it by changing my local copy of cptable.js and it works after the change.
The way I read the documentation, the tables are auto generated from some references on the web, so I did not bother to generate a pull request, since I do not want to mess with your auto generation and have only little time to look further into it or fix more than codepage 850.

Maybe you can fix it in the auto generation? If not, let me know and I will at least make a pull-request for cp850, once I have manually fixed all the characters in this one.

All the best!
Thomas

P.S.: Wikipedia has the correct translation:
https://de.wikipedia.org/wiki/Codepage_850