microsoft/vscode-nls-dev

The tools automates the extraction of strings to be externalized from TS and JS code. It therefore help localizing VSCode extensions and language servers written in TS and JS

vscode-nls-dev

⚠️ This package is no longer receiving new features in favor of the new localization library, vscode-l10n. Please use that collection of libraries instead.

The tools automates the extraction of strings to be externalized from TS and JS code. It therefore helps localizing VSCode extensions and language servers written in TS and JS. It also contains helper methods to convert unlocalized JSON to XLIFF format for translations, and back to localized JSON files, with ability to push and pull localizations from Transifex platform.

4.0.1

• Allow overriding of baseDir in proccessing files.

4.0.0-next.1

• Add support for comments in messages (e.g. package.nls.json)
• Remove Transifex support
• General code cleanup. Move to TS 4.3.1 and more stricter type checking.

3.3.2

• Merged allow es imports, update ts and use their helper methods

3.0.0

• added support to bundle the strings into a single nls.bundle(.${locale})?.json file.
• added support for VS Code language packs.

2.1.0:

• Add support to push to and pull from Transifex.

2.0.0:

• based on TypeScript 2.0. Since TS changed the shape of the d.ts files for 2.0.x a major version number got introduce to not break existing clients using TypeScript 1.8.x.

JSON->XLIFF->JSON

To perform unlocalized JSON to XLIFF conversion it is required to call prepareXlfFiles(projectName, extensionName) piping your extension/language server directory to it, where projectName is the Transifex project name (if such exists) and extensionName is the name of your extension/language server. Thereby, XLF files will have a path of projectName/extensionName.xlf.

To convert translated XLIFF to localized JSON files prepareJsonFiles() should be called, piping .xlf files to it. It will parse translated XLIFF to JSON files, reconstructed under original file paths.

Transifex Push and Pull

Updating Transifex with latest unlocalized strings is done via pushXlfFiles('www.transifex.com', apiName, apiToken) and pullXlfFiles('www.transifex.com', apiName, apiToken, languages, resources) for pulling localizations respectively. When pulling, you have to provide resources array with object literals that have name and project properties. name corresponds to the resource name in Transifex and project is a project name of your Transifex project where this resource is stored. languages argument is an array of strings of culture names to be pulled from Transifex.

Onboarding Extension to Transifex

Here is a sample code that adds localization using Transifex. You can copy and use it as a template for your own extension, changing the values to the ones described in the code comments.

var nls = require('vscode-nls-dev');
const vscodeLanguages = [
	'zh-hans',
	'zh-hant',
	'ja',
	'ko',
	'de',
	'fr',
	'es',
	'ru',
	'it'
]; // languages an extension has to be translated to

const transifexApiHostname = 'www.transifex.com';
const transifexApiName = 'api';
const transifexApiToken = process.env.TRANSIFEX_API_TOKEN; // token to talk to Transifex (to obtain it see https://docs.transifex.com/api/introduction#authentication)
const transifexProjectName = 'vscode-extensions'; // your project name in Transifex
const transifexExtensionName = 'vscode-node-debug'; // your resource name in Transifex

gulp.task('transifex-push', function() {
	return gulp.src('**/*.nls.json')
		.pipe(nls.prepareXlfFiles(transifexProjectName, transifexExtensionName))
		.pipe(nls.pushXlfFiles(transifexApiHostname, transifexApiName, transifexApiToken));
});

gulp.task('transifex-pull', function() {
	return nls.pullXlfFiles(transifexApiHostname, transifexApiName, transifexApiToken, vscodeLanguages, [{ name: transifexExtensionName, project: transifexProjectName }])
		.pipe(gulp.dest(`../${transifexExtensionName}-localization`));
});

gulp.task('i18n-import', function() {
	return gulp.src(`../${transifexExtensionName}-localization/**/*.xlf`)
		.pipe(nls.prepareJsonFiles())
		.pipe(gulp.dest('./i18n'));
});

To push strings for translation to Transifex you call gulp transifex-push. To pull and perform the import of latest translations from Transifex to your extension, you need to call transifex-pull and i18n-import sequentially. This will pull XLF files from Transifex in first gulp task, and import them to i18n folder in JSON format.

LICENSE

MIT

vscode-nls-dev's Issues

Typescript esModuleInterop incompatibility issue

Hi. Using the exact sample project here: https://github.com/microsoft/vscode-extension-samples/tree/master/i18n-sample, if I add "esModuleInterop": true to the compilerOptions in tsconfig.json, then language files for all typescript files are not generated when running gulp build. Is this expected behaviour? I'd like to use esModuleInterop for importing 3rd party packages.

It's not clear if i should post this issue in https://github.com/microsoft/vscode-nls instead.

Fails on ES6 backtick quotes

We have this code that vscode-nls-dev rewrites:

public static failedToEnumerateFilesInDir(path: string, errCode: string, errMessage: string): string {
    return localize(
        {
            key: 'Failed.To.Enumerate.Files.In.Dir',
            comment:
            [
                'Error message when failed to enumerate files in a directory.'
            ]
        },
        `Failed to enumerate files in directory '{0}'. Error code={1}, Error message={2}`,
        path,
        errCode,
        errMessage
    );
}

When targeting ES5, all is well. But once we target ES6, the backtick as a quote character is preserved:

static failedToEnumerateFilesInDir(path, errCode, errMessage) {
    return localize({
        key: 'Failed.To.Enumerate.Files.In.Dir',
        comment: [
            'Error message when failed to enumerate files in a directory.'
        ]
    }, `Failed to enumerate files in directory '{0}'. Error code={1}, Error message={2}`, path, errCode, errMessage);
}

This causes vscode-nls-dev to fail during build with this error:

libraryResourceStrings.js(661,11): second argument of a localize call must be a string literal.

For now we workaround it by changing our own use of backtick to single quotes. But that requires much more escaping in our English strings because we use single-quotes inside the localizable string.

Please enable use of backticks.

Pre-release pipelines failing with `host.fileExists is not a function`

vscode-nls-dev version 4.0.1

Module build failed (from ./node_modules/vscode-nls-dev/lib/webpack-loader.js):
TypeError: host.fileExists is not a function
    at Object.fileExists (/mnt/vss/_work/1/s/node_modules/typescript/lib/typescript.js:167586:63)
    at Object.host.fileExists (/mnt/vss/_work/1/s/node_modules/typescript/lib/typescript.js:117448:47)
    at tryFileLookup (/mnt/vss/_work/1/s/node_modules/typescript/lib/typescript.js:44293:28)
    at tryFile (/mnt/vss/_work/1/s/node_modules/typescript/lib/typescript.js:44285:20)
    at tryExtension (/mnt/vss/_work/1/s/node_modules/typescript/lib/typescript.js:44277:24)
    at tryAddingExtensions (/mnt/vss/_work/1/s/node_modules/typescript/lib/typescript.js:44255:32)
    at loadModuleFromFile (/mnt/vss/_work/1/s/node_modules/typescript/lib/typescript.js:44188:45)
    at nodeLoadModuleByRelativeName (/mnt/vss/_work/1/s/node_modules/typescript/lib/typescript.js:44112:36)
    at tryResolve (/mnt/vss/_work/1/s/node_modules/typescript/lib/typescript.js:44069:34)
    at /mnt/vss/_work/1/s/node_modules/typescript/lib/typescript.js:44032:69

ref: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=184047&view=logs&j=eb79cae4-0ffb-5516-9c82-40a537d0707b&t=c0e8a551-bf7a-588d-e12e-b60cae8f30b8

createKeyValuePairFile should provide options to join comments into one string

Please provide instructions how to initialize i18n files without Transifex

Hi,

as I understand this tool can take existing vscode extension (where all messages are directly in package.json and in .ts files) and extract the messages to package.nls.json and to files inside i18n, replacing the messages with %placeholders% and locallize().

It can also push/pull messages from the Transifex platform, but it does not need the platform to extract the original messages in the above "init" step.

If that's true, could you please provide instructions how to use this tool to init the i18n files without the Transifex platform?

Thank you.

Need a way to communicate translation hints

Similar to #20 , we would like to be able to provide notes/hints for translators. Perhaps hints could be passed directly to the call to localize() for inclusion in exported xlf files.

One possible approach might be an alternate localize() function which accepts an additional string for notes/hints. i.e. localize_with_hint(key, hint, localizableString, ...stringArgs)

This would seem to require a change to the format of the exported JSON, or perhaps an additional exported file containing a mapping of hints to string keys, which can be read when generating the xlf for export.

Allow createXlfFiles() respect existing <note> on xlf file.

We use createXlfFiles to generate ENU xlf file from 'package.nls.json', 'nls.metadata.header.json', and 'nls.metadata.json' files. Our loc team uses this ENU xlf file to produce translated xlfs.

The issue is that we have to have <note> elements in the trans units in our ENU xlf to pass instructions to the loc team, and createXlfFiles() just overwrites the ENU xlf, losing all the notes.

We had to work it around by getting the notes from the existing ENU xlf file first, and then applying them in our copy of createXlfFiles() as in this PR

I think this is a common scenario to be enabled in vscode-nls-dev

Unable to bundle library which uses nls

Hey, I trying to create a plugin on top of vscode-chrome-debug-core, which uses nls to handle translations. When I bundle my plugin with webpack using nls-dev bundler and loader, it does not import translations of vscode-chrome-debug-core dependency. As a result, I see "Failed to load message bundle..." message in my extension.

Is it possible to bundle dependency translations as part of my extension?

const config = {
  target: 'node',

  node: {
    __dirname: false
  },

  entry: './src/extension.ts',

  output: {
    path: path.resolve(__dirname, 'dist'),
    filename: 'extension.js',
    libraryTarget: 'commonjs2',
    devtoolModuleFilenameTemplate: '../[resource-path]',
  },

  devtool: 'source-map',

  externals: [{
    vscode: 'commonjs vscode',
  }],

  resolve: {
    extensions: ['.ts', '.js'],
    plugins: [
      new TsconfigPathsPlugin({
        configFile: path.resolve(__dirname, '..', '..', 'tsconfig.json'),
      }),
    ],
  },

  plugins: [
    new NLSBundlePlugin('reactvr-vscode-plugin'),
  ],

  module: {
    rules: [
      {
        test: /\.ts$/,
        exclude: /node_modules/,
        use: [{
					loader: 'vscode-nls-dev/lib/webpack-loader',
					options: {
						base: path.join(__dirname, 'src')
					}
				}, {
					loader: 'ts-loader',
				}]
      },
    ],
  },
};

Fail Compile TypeScript

On compile typescript generate error:

[failed] [1 error]

ERROR in ./src/extension.ts 22:21
Module parse failed: Unexpected token (22:21)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
| const localize = nls.config({ messageFormat: nls.MessageFormat.file })();
|

let pendingValidation: boolean = false;
|
| let projeto: ValidaProjeto;

My webpack.config.js

/*---------------------------------------------------------------------------------------------
 *  Copyright (c) Microsoft Corporation. All rights reserved.
 *  Licensed under the MIT License. See License.txt in the project root for license information.
 *--------------------------------------------------------------------------------------------*/

//@ts-check

'use strict';

const path = require('path');

/**@type {import('webpack').Configuration}*/
const config = {
  target: 'node', // vscode extensions run in a Node.js-context 📖 -> https://webpack.js.org/configuration/node/

  entry: './src/extension.ts', // the entry point of this extension, 📖 -> https://webpack.js.org/configuration/entry-context/
  output: { // the bundle is stored in the 'dist' folder (check package.json), 📖 -> https://webpack.js.org/configuration/output/
    path: path.resolve(__dirname, 'dist'),
    filename: 'extension.js',
    libraryTarget: "commonjs2",
    devtoolModuleFilenameTemplate: "../[resource-path]",
  },
  devtool: 'source-map',
  externals: {
    vscode: "commonjs vscode" // the vscode-module is created on-the-fly and must be excluded. Add other modules that cannot be webpack'ed, 📖 -> https://webpack.js.org/configuration/externals/
  },
  resolve: { // support reading TypeScript and JavaScript files, 📖 -> https://github.com/TypeStrong/ts-loader
    extensions: ['.js', '.ts', '.tsx']
  },
  module: {
    rules: [
      {
        test: /\.\.\/*$/,
        exclude: /node_modules/,
        use: [
          {
            loader: 'vscode-nls-dev/lib/webpack-loader',
            options: {
              base: path.join(__dirname, 'src')
            }
          }
        ]
      }
    ]
  },
}

module.exports = config;
`

Update strip-ansi dependency to fix security warning

To fix security warning described: https://github.com/microsoft/vscode-remote-ssh/security/dependabot/open-ssh-remote/yarn.lock/ansi-regex/open

Allow to select the source-language in createXlfFiles()

Hi,
right now createXlfFiles() generates a xlf file where source-language is hardcoded to "en"
Is it possible to add the possibility to specify the source-language in case someone wants to create the main language strings in a different language?

Thanks in advance

Need a way to provide a comment/hint in package.nls.json

This is a request to provide support for specifying a comment/hint in package.nls.json, for translators

We'd like to start using the markdownDescription field for some of our extension settings. We'd like to be able to provide a comment/hint to translators that these strings (in package.nls.json) contain markdown, so translators are aware of it and avoid removing or corrupting the markdown syntax.

I don't see any comment/hint field handled for package.nls.json, in the implementation of createXlfFiles(). I see that it supports specifying an object with a message field instead of a string field. But, I don't see anywhere that it also handles a comment/hint field in that object.

All localized strings show up as 'Failed to load message bundle' in extension development host

E.g. for RemoteHub:

Import syntax matters

From my testing, the import syntax used had to be in the form:

import * as nls from 'vscode-nls';

and the following wasn't picked up:

import { loadMessageBundle } from 'vscode-nls';

rewriteLocalizeCalls is messing up sourcemappings in vscode-node-debug

In nodeDebug.ts, this comment exists at line 431:

/*
this._node.on('beforeCompile', (event: NodeV8Event) => {
	this.outLine(`beforeCompile ${this._scriptToPath(event.body.script)}`);
});

this._node.on('afterCompile', (event: NodeV8Event) => {
	this.outLine(`afterCompile ${this._scriptToPath(event.body.script)}`);
});
*/

After this comment, the sourcemappings are incorrect, making it very hard to debug. You can see this by building and using the sourcemap visualizer: https://sokra.github.io/source-map-visualization. There shouldn't be several mappings on these lines, on individual letters, etc:

It looks like the empty line between those two blocks is removed, and the subsequent mappings are all off by a line. Looking at the code, I have no clue why that would happen.

Update to xml2js from 4.23 to 0.5.0

This needs to be done for security purposes in projects that use it.

Show locale when warning on missing translations

When translations are missing, the errors do not include the language for which the translations are missing. This makes all errors look the same and it is rather difficult to understand what is really missing.

In the following screenshot it is difficult to spot that 10 strings are missing translations for all languages:

Suppress unnecessary error messages

In my node-debug extension I have a utilities.ts with this contents:

'use strict';

import * as vscode from 'vscode';
import * as nls from 'vscode-nls';

export const localize = nls.config(process.env.VSCODE_NLS_CONFIG)();

export function log(message: string) {
	vscode.debug.activeDebugConsole.appendLine(message);
}

When building I always get these unhelpful error messages:

[12:10:42] [i18n] Generating localized messages for 'chs' resulted in the following problems:
[12:10:42] [i18n]
[12:10:42] [i18n] Message file chs/out/node/extension/utilities.i18n.json not found. Missing messages: 0
[12:10:42] [i18n]
[12:10:42] [i18n]
[12:10:42] [i18n] Generating localized messages for 'cht' resulted in the following problems:
[12:10:42] [i18n]
[12:10:42] [i18n] Message file cht/out/node/extension/utilities.i18n.json not found. Missing messages: 0
[12:10:42] [i18n]
[12:10:42] [i18n]
[12:10:42] [i18n] Generating localized messages for 'jpn' resulted in the following problems:
[12:10:42] [i18n]
[12:10:42] [i18n] Message file jpn/out/node/extension/utilities.i18n.json not found. Missing messages: 0
[12:10:42] [i18n]
[12:10:42] [i18n]
[12:10:42] [i18n] Generating localized messages for 'kor' resulted in the following problems:
[12:10:42] [i18n]
[12:10:42] [i18n] Message file kor/out/node/extension/utilities.i18n.json not found. Missing messages: 0
[12:10:42] [i18n]
[12:10:42] [i18n]
[12:10:42] [i18n] Generating localized messages for 'deu' resulted in the following problems:
[12:10:42] [i18n]
[12:10:42] [i18n] Message file deu/out/node/extension/utilities.i18n.json not found. Missing messages: 0
[12:10:42] [i18n]
[12:10:42] [i18n]
[12:10:42] [i18n] Generating localized messages for 'fra' resulted in the following problems:
[12:10:42] [i18n]
[12:10:42] [i18n] Message file fra/out/node/extension/utilities.i18n.json not found. Missing messages: 0
[12:10:42] [i18n]
[12:10:42] [i18n]
[12:10:42] [i18n] Generating localized messages for 'esn' resulted in the following problems:
[12:10:42] [i18n]
[12:10:42] [i18n] Message file esn/out/node/extension/utilities.i18n.json not found. Missing messages: 0
[12:10:42] [i18n]
[12:10:42] [i18n]
[12:10:42] [i18n] Generating localized messages for 'rus' resulted in the following problems:
[12:10:42] [i18n]
[12:10:42] [i18n] Message file rus/out/node/extension/utilities.i18n.json not found. Missing messages: 0
[12:10:42] [i18n]
[12:10:42] [i18n]
[12:10:42] [i18n] Generating localized messages for 'ita' resulted in the following problems:
[12:10:42] [i18n]
[12:10:42] [i18n] Message file ita/out/node/extension/utilities.i18n.json not found. Missing messages: 0
[12:10:42] [i18n]
[12:10:42] [i18n]

Yes, there is no "utilities.i18n.json" but there are no missing translated strings either.
It would be great if these unnecessary errors could be suppressed.

The default branch has been renamed from 'master' to 'main'

⚠️ Call to Action ⚠️

If you have a clone of this repository locally, you'll need to run the following commands to catch-up with the rename

git branch -m master main
git fetch origin
git branch -u origin/main main

Sourcemaps lose their inline content when going through rewriteLocalizeCalls

We need to do the same dance done here:

• https://github.com/Microsoft/vscode/blob/master/build/lib/nls.ts#L376
• https://github.com/Microsoft/vscode/blob/master/build/lib/nls.ts#L382

backslashes to enable multi-line string literals get exported

A call to localize() such as the following:

localize('example', 'example \
example \
example')

... results in the backslashes being included in the exported string. However, these backslashes are intended to allow the literal to be multi-line, and are not intended to be part of the string.

File paths should be absolute when emitting errors

From microsoft/vscode#48393

Make locale tag resolution lower case based

For all the details see microsoft/vscode#4520

Update documentation with info on onboarding extensions to Transifex

rewriteLocalizeCalls() doesn't work with 'importHelpers = true'

When importHelpers is set to true in tsconfig.json, rewriteLocalizeCalls() doesn't generate any bundle files so all localize() invocations go to fallback.

Did a little debug and found out that, the condition here

vscode-nls-dev/src/lib.ts

Line 472 in 545e01a

if (ts.isCallExpression(parent) && ts.isIdentifier(parent.expression) && parent.expression.text === '__importStar') {

is not met.

Possible JS transpiled when using tslib import helpers:

const nls = tslib_1.__importStar(require("vscode-nls"));

Here the expression turns from a global function call to a member access. This case can be handled by (for example):

if (ts.isCallExpression(parent) && ((ts.isIdentifier(parent.expression) && parent.expression.text === '__importStar') || (ts.isPropertyAccessExpression(parent.expression) && parent.expression.name.text === "__importStar")))

XLF.parse() fails to extract target text if <target> element has "state" attribute

If XLF.parse() is used to parse the following xliff xml, it'll return [Object object] instead of proper translations:

<xliff version="1.2" xmlns="urn:oasis:names:tc:xliff:document:1.2">
  <file original="package" source-language="en" target-language="zh-Hant" datatype="plaintext"><body>
    <trans-unit id="extension.liveshare.start.title">
      <source xml:lang="en">Start Collaboration Session (Share)</source>
      <target state="translated">開始共同作業工作階段 (共用)</target>
    </trans-unit>
...

The reason is that if the <target> element has state attribute, then unit.target.toString() returns '[Object object]' on line 591 in main.ts

`TypeError: sourceMapConsumer.eachMapping is not a function` with `"source-map": "^0.8.0-beta.0"`

Our CI lab upgraded Node to v18 on our CI VM's. That caused us to hit an issue (discussed here: parcel-bundler/parcel#8005 )

The suggested work-around for that issue helped. (to add "source-map": "^0.8.0-beta.0" to the resolutions section).

Unfortunately, this fixed version of source-map appears to break compatibility with vscode-nls-dev. We're now getting:

ERROR in ./src/main.ts
Module build failed (from ./node_modules/vscode-nls-dev/lib/webpack-loader.js):
TypeError: sourceMapConsumer.eachMapping is not a function
    at new TextModel (Z:\repos\vscode-cpptools\Extension\node_modules\vscode-nls-dev\lib\lib.js:116:31)
    at Object.processFile (Z:\repos\vscode-cpptools\Extension\node_modules\vscode-nls-dev\lib\lib.js:572:23)
    at module.exports (Z:\repos\vscode-cpptools\Extension\node_modules\vscode-nls-dev\lib\webpack-loader.js:17:26)

When bundling, *.nls.metadata.json files are kept around

These files shouldn't be included in the destination and are intermidiate.