motss/normalize-diacritics

Normalize diacritics in strings

normalize-diacritics

Remove accents/ diacritics in string

---

Simple NPM package to remove any accents/ diacritics found in a string.

Table of contents

• Pre-requisite
• Install
• Usage
  • TypeScript or ES Modules
  • Browser
• Demo
• deno
• API Reference
  • normalize([input])
  • normalizeSync([input])
• Contributing
  • Code of Conduct
• License

Pre-requisite

• Node.js >= 14.17.3
• NPM >= 6.14.13 (NPM comes with Node.js so there is no need to install separately.)
• ES Modules

Install

# Install via NPM
$ npm i normalize-diacritics

Usage

TypeScript or ES Modules

import { normalize } from 'normalize-diacritics';

/** Assuming top-level await is enabled... */
await normalize('söme stüff with áccènts'); // 'some stuff with accents'

Browser

<script type="module">
  import { normalize } from 'https://cdn.skypack.dev/normalize-diacritics@latest';

  // --snip
</script>

Demo

deno

👉 Check out the deno module at deno_mod/normalize_diacritics.

API Reference

normalize([input])

• input <string> Input string that contains accents/diacritics.
• returns: <Promise<string>> Promise which resolves with normalized input string.

This method normalizes any accents/ diacritics found in a given input string and output a normalized string as a result.

normalizeSync([input])

This methods works the same as normalize([input]) except that this is the synchronous version.

Contributing

Code of Conduct

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

License

MIT License © Rong Sen Ng

normalize-diacritics's Issues

Bug due to RegExp lastIndex state

Repeated letters with the same diacritic are not consistently normalized. E.g.:

> normalizeSync('éééé')
'eéeé'
> normalizeSync('åååå')
'aåaå'
> normalizeSync('éåéåéåéå')
'eaéaeaéa'

(Live demo: https://repl.it/repls/LeftFirsthandDatabases)

This bug is caused by the use of RegExp.prototype.test (normalize-diacritics.ts#L131), which keeps state when called on a regex that has the /global flag;

If the regex has the global flag set, test() will advance the lastIndex of the regex. A subsequent use of test() will start the search at the substring of str specified by lastIndex

Source: MDN.

To use Jest as testing tool

Jest seems to have better support for TypeScript than that of AVA. This means that code coverage can make a comeback too! 👍

No typing file

rollup-plugin-typescript2 should do the trick.

Typo in README

- NPM come Node.js

Ready to use @messageflow/clean-cli

Description

Expected outcome

Actual outcome

Live Demo

Steps to reproduce

Versions of affected system

• normalize-diacritics version:
• Node.js version:
• Platform details (e.g. macOS Sierra 10.12.4):

To provider better linting command with gulp lint

CommonJS require fails because of improper `exports` order

Description

CommonJS (cjs) import no longer works. Try from a node v14+ REPL

> const nd = require("normalize-diacritics")
Uncaught:
Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: 

Expected outcome

No error on CJS import.

Actual outcome

ERR_REQUIRE_ESM

Fix

Per conditional export docs, order matters: place "default": "./dist/index.mjs" at the end of the list and everything works!

Versions of affected system

• normalize-diacritics version: 2.13.1
• Node.js version: 14.15.1

Promote normalize_diacritics for deno

TO-DO

• Show some ❤️ to deno users

Update project structure for TypeScript refactoring

Doesnt normalize Danish ø

Description

The normalize-diacritics does not work correctly with danish ø
When running it on tromsø

Expected outcome

tromso

Actual outcome

tromsø

Live Demo

n/a

Steps to reproduce

Versions of affected system

• normalize-diacritics version: 2.2.3
• Node.js version: 10.11
• Platform details (macOS Sierra 10.13.6):

To update deps

To use prepublishOnly for publish package

To clarify instructions on using different bundle

Postinstall error on ZSH console

Description

Package postinstall script fails in ZSH console when installing the dependency through npm

Expected outcome

Package successfully installed

Actual outcome

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] postinstall: bash postinstall.bash
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Steps to reproduce

1. Run npm install normalize-diacritics.

-->

Versions of affected system

• normalize-diacritics version: 3.0.6
• Node.js version: v14.15.1
• Platform details: macOS Big Sur 11.6.4

An in-range update of nsp is breaking the build 🚨

Version 2.8.0 of nsp just got published.

Branch	Build failing 🚨
Dependency	nsp
Current Version	2.7.0
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

As nsp is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Status Details

• ❌ coverage/coveralls Coverage pending from Coveralls.io Details
• ✅ continuous-integration/travis-ci/push The Travis CI build passed Details
• ❌ continuous-integration/appveyor/branch AppVeyor build failed Details

---

Commits

The new version differs by 7 commits.

• 93816b2 2.8.0
• 6dfcca4 shrinkwrap shuffle apparently
• 31029c2 Merge pull request #184 from nodesecurity/lock
• fb48dd1 Merge branch 'master' of github.com:nodesecurity/nsp into lock
• 82ee301 remove code don't comment it out
• 1d0b7b5 adding package-lock.json support
• 56f9848 Fix build status and add mention of nodesecurity.io

See the full diff

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.

---

Your Greenkeeper Bot 🌴

An in-range update of babel-cli is breaking the build 🚨

Version 6.26.0 of babel-cli just got published.

Branch	Build failing 🚨
Dependency	babel-cli
Current Version	6.24.1
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

As babel-cli is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Status Details

• ✅ continuous-integration/travis-ci/push The Travis CI build passed Details
• ✅ coverage/coveralls First build on greenkeeper/babel-cli-6.26.0 at 100.0% Details
• ❌ continuous-integration/appveyor/branch AppVeyor build failed Details

---

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.

---

Your Greenkeeper Bot 🌴

An in-range update of snyk is breaking the build 🚨

Version 1.34.1 of snyk just got published.

Branch	Build failing 🚨
Dependency	snyk
Current Version	1.34.0
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

As snyk is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Status Details

• ❌ continuous-integration/appveyor/branch Waiting for AppVeyor build to complete Details
• ❌ continuous-integration/travis-ci/push The Travis CI build failed Details
• ✅ coverage/coveralls First build on greenkeeper/snyk-1.34.1 at 100.0% Details

---

Release Notes

v1.34.1

<a name"1.34.1">

1.34.1 (2017-06-06)

Bug Fixes

• remove 'runtime' as default for gradle, auto-detect gradle, support omitted depe (960f07e8)

Commits

The new version differs by 2 commits.

• 92429bb chore: PR comments
• 960f07e fix: remove 'runtime' as default for gradle, auto-detect gradle, support omitted dependencies in gradle

See the full diff

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.

---

Your Greenkeeper Bot 🌴

why dedupe?

why does this package run npm dedupe after install? i don't even use npm, this adds unnecessary delay when installing dependencies. i know you can disable postinstall scripts but i still think it should be removed from the package

To update .gitignore

- test*/
+ ./test*/

Proper implementation in ES6

https://stackoverflow.com/questions/990904/remove-accents-diacritics-in-a-string-in-javascript/37511463#37511463

An in-range update of codecov is breaking the build 🚨

Version 2.3.0 of codecov just got published.

Branch	Build failing 🚨
Dependency	codecov
Current Version	2.2.0
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

As codecov is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Status Details

• ❌ coverage/coveralls Coverage pending from Coveralls.io Details
• ✅ continuous-integration/appveyor/branch AppVeyor build succeeded Details
• ❌ continuous-integration/travis-ci/push The Travis CI build could not complete due to an error Details

---

Commits

The new version differs by 6 commits.

• 7e39f19 v2.3.0
• 489ec49 Adding AppVeyor badge
• 8e31537 Coverage upload on Windows - fixes #61 (#62)
• 12a5193 Update package.json (#67)
• 2e40593 Update README.md
• d2a021d Update README.md

See the full diff

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.

---

Your Greenkeeper Bot 🌴

To add jsdelivr CDN URL in deno branch

https://cdn.jsdelivr.net/gh/motss/[email protected]/index.ts

bug: postinstall fails in pnpm-monorepo

The "postinstall" is specific to working on the package itself and not useful for consumers of the package.

Consider "prepare" instead? or maybe there is a better match: https://docs.npmjs.com/cli/v8/using-npm/scripts#life-cycle-scripts. You can also consider a custom bootstrap script for when you need to work on the package.

Description

postinstall fails in a pnpm monorepo

Expected outcome

That normalize-diacritics installs correctly

Actual outcome

<my-user> and <our-repo> is redacted information

...
node_modules/.pnpm/[email protected]/node_modules/normalize-diacritics: Running postinstall script, failed in 481ms
.../node_modules/normalize-diacritics postinstall$ sh postinstall.sh
│ npm ERR! code ENOENT
│ npm ERR! syscall lstat
│ npm ERR! path /home/<my-user>/work/<our-repo>/lib
│ npm ERR! errno -2
│ npm ERR! enoent ENOENT: no such file or directory, lstat '/home/<my-user>/work/<our-repo>/lib'
│ npm ERR! enoent This is related to npm not being able to find a file.
│ npm ERR! enoent
│ npm ERR! A complete log of this run can be found in:
│ npm ERR!     /home/<my-user>/.npm/_logs/2022-11-08T09_59_58_504Z-debug-0.log
...

The line that triggers the error:
postinstall.sh

npx -y -- zx@latest ./postinstall.mjs

Steps to reproduce

We run (in our monorepo)
<repo>/package.json

    "workspace--install": "git config  url.\"ssh://git@\".insteadOf git:// && pnpm recursive install --link-workspace-packages",

Versions of affected system

• package: [email protected]
• pnpm --version => 7.14.0
• nix --version => nix (Nix) 2.11.1
• platform: Linux <my-hostname> 5.15.0-52-generic #58-Ubuntu SMP Thu Oct 13 08:03:55 UTC 2022 x86_64 GNU/Linux

To remove codecov, coveralls, and greenkeeper

No longer need them for TypeScript.

To restore String#normalize when deno ships with ICU

Related issues:

• denoland/deno#1636
• denoland/deno#1952

Question: Why an async version that is not doing any async stuff?

I don't get it.

What is the reason of having this? In which scenario could be better to call this function instead of normaliceSync ?

export async function normalize(input?: string | null) {
  return normalizeSync(input);
}

SyntaxError: Unexpected token 'export'

Description

This module does not seem to be friend with Jest in a expo project with ESM modules.

I do know possess the knowledge to establish what is going wrong, but I am posting this for future developers who encounter this issue.

//jest.config.js
/** @type {import('ts-jest').JestConfigWithTsJest} */
export default {
  'preset': 'jest-expo',
  testEnvironment: 'node',
};

Expected outcome

Should work.

Actual outcome

    Jest encountered an unexpected token

    Jest failed to parse a file. This happens e.g. when your code or its dependencies use non-standard JavaScript syntax, or when Jest is not configured to support such syntax.

    Out of the box Jest supports Babel, which will be used to transform your files into valid JS based on your Babel configuration.

    By default "node_modules" folder is ignored by transformers.

    Here's what you can do:
     • If you are trying to use ECMAScript Modules, see https://jestjs.io/docs/ecmascript-modules for how to enable it.
     • If you are trying to use TypeScript, see https://jestjs.io/docs/getting-started#using-typescript
     • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
     • If you need a custom transformation specify a "transform" option in your config.
     • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

    You'll find more details and examples of these config options in the docs:
    https://jestjs.io/docs/configuration
    For information about custom transformations, see:
    https://jestjs.io/docs/code-transformation

    Details:

    /home/user/project/node_modules/normalize-diacritics/dist/index.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,jest){export * from './normalize.js';
                                                                                      ^^^^^^

    SyntaxError: Unexpected token 'export'

      1 | import foobar from 'foobar';
      2 | import { foobar } from './foobar';
    > 3 | import { normalizeSync } from 'normalize-diacritics';
        | ^
      4 | import { foobar } from '../foobar';
      5 | import { foobar } from './foobar';
      6 | import { foobar } from './foobar';

      at Runtime.createScriptFromCode (node_modules/jest-runtime/build/index.js:1505:14)
      at Object.require (app/foobar/foobar.ts:3:1)
      at Object.require (app/foobar/foobar.test.ts:1:1)

Workaround

Use https://github.com/andrewrk/node-diacritics instead, which seems to work fine with Jest.

The letter ß (Eszett) should be normalized to ss

Description

The letter ß should be normalized to ss and not s => https://en.wikipedia.org/wiki/%C3%9F

Expected outcome

normalizeSync("Muße") => "musse"

Actual outcome

normalizeSync("Muße") => "muse"

Versions of affected system

• 4.0.0

To add description in returns

An in-range update of nyc is breaking the build 🚨

Version 11.2.0 of nyc just got published.

Branch	Build failing 🚨
Dependency	nyc
Current Version	11.1.0
Type	devDependency

This version is covered by your current version range and after updating it in your project the build failed.

As nyc is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Status Details

• ✅ continuous-integration/travis-ci/push The Travis CI build passed Details
• ✅ coverage/coveralls First build on greenkeeper/nyc-11.2.0 at 100.0% Details
• ❌ continuous-integration/appveyor/branch AppVeyor build failed Details

---

Commits

The new version differs by 7 commits.

• b2ff72d chore(release): 11.2.0
• 97b962e chore: explicit update of istanbul-lib-instrument (#662)
• 0dcceda feat: allow cwd to be configured see #620
• 0fc6d8f chore: add support for --cwd (#542) (#620)
• 658dba4 fix: remove excluded files from coverage before writing (#649)
• dd40dc5 feat: add possibility to filter coverage-maps (#637)
• fb3ab92 docs: add link to community slack (#639)

See the full diff

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.

---

Your Greenkeeper Bot 🌴

To update to a better tslint.json that is in the works

To ship .js for Deno users

Description

As pointed out in https://github.com/denoland/registry/issues/39, deno can run both JS and TS. For JavaScript users, they would expect .js file of all modules, not just .ts.

Related issues

• denoland/registry#79

Consider limiting `input` type to `string`

Description

The type for parameter input in normalizeSync implies that undefined and null are acceptable inputs. However, the first line of the function throws an error if the input is not a string.

https://github.com/motss/normalize-diacritics/blob/main/src/normalize-sync.ts#L3-L6

I believe it would be more ergonomic to have TypeScript signal to the dev that only strings are acceptable.
In my opinion, TS is best suited to help devs avoid runtime errors by attempting to move common issues to compile-time errors. With this in mind, I think it would be more helpful to set input to only string.

export function normalizeSync(input: string): string {
  //....
}

That way we get type errors before running the code:

/** @type {string | undefined | null} */
const input = getInputFromWherever();

normalizeSync(input);
              ^^^^
              // Argument of type 'string | null | undefined' 
              // is not assignable to parameter of type 'string'.
              // Type 'undefined' is not assignable to type 'string'.ts(2345)

What do you think?

Anyway, thanks for building this lib 🙂

No normalization when input is a 1-character string.