bootphon/phonemizer

Simple text to phones converter for multiple languages

Tests
Documentation
Release
Citation

---

Phonemizer -- foʊnmaɪzɚ

• The phonemizer allows simple phonemization of words and texts in many languages.

• Provides both the phonemize command-line tool and the Python function phonemizer.phonemize. See the package's documentation.

• It is based on four backends: espeak, espeak-mbrola, festival and segments. The backends have different properties and capabilities resumed in table below. The backend choice is let to the user.

  • espeak-ng is a Text-to-Speech software supporting a lot of languages and IPA (International Phonetic Alphabet) output.

  • espeak-ng-mbrola uses the SAMPA phonetic alphabet instead of IPA but does not preserve word boundaries.

  • festival is another Tex-to-Speech engine. Its phonemizer backend currently supports only American English. It uses a custom phoneset, but it allows tokenization at the syllable level.

  • segments is a Unicode tokenizer that build a phonemization from a grapheme to phoneme mapping provided as a file by the user.

  	espeak	espeak-mbrola	festival	segments
  phone set	IPA	SAMPA	custom	user defined
  supported languages	100+	35	US English	user defined
  processing speed	fast	slow	very slow	fast
  phone tokens	✔️	✔️	✔️	✔️
  syllable tokens	❌	❌	✔️	❌
  word tokens	✔️	❌	✔️	✔️
  punctuation preservation	✔️	❌	✔️	✔️
  stressed phones	✔️	❌	❌	❌
  tie	✔️	❌	❌	❌

Citation

To refenrece the phonemizer in your own work, please cite the following JOSS paper.

@article{Bernard2021,
  doi = {10.21105/joss.03958},
  url = {https://doi.org/10.21105/joss.03958},
  year = {2021},
  publisher = {The Open Journal},
  volume = {6},
  number = {68},
  pages = {3958},
  author = {Mathieu Bernard and Hadrien Titeux},
  title = {Phonemizer: Text to Phones Transcription for Multiple Languages in Python},
  journal = {Journal of Open Source Software}
}

Licence

Copyright 2015-2021 Mathieu Bernard

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

phonemizer's Issues

Failure with preserve_punctuation=True

from phonemizer.phonemize import phonemize
phonemize("hi; ho,\"", backend="espeak", preserve_punctuation=True, punctuation_marks=".!;:,?")

Traceback (most recent call last):
  File "E:\Python37\lib\site-packages\IPython\core\interactiveshell.py", line 3291, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-26-f9fe7d23f4b3>", line 1, in <module>
    phonemize("hi; ho,\"", backend="espeak", preserve_punctuation=True, punctuation_marks=".!;:,?")
  File "E:\Python37\lib\site-packages\phonemizer\phonemize.py", line 172, in phonemize
    text, separator=separator, strip=strip, njobs=njobs)
  File "E:\Python37\lib\site-packages\phonemizer\backend\base.py", line 126, in phonemize
    text = self._punctuator.restore(text, punctuation_marks)
  File "E:\Python37\lib\site-packages\phonemizer\punctuation.py", line 146, in restore
    return cls._restore_aux(str2list(text), marks, 0)
  File "E:\Python37\lib\site-packages\phonemizer\punctuation.py", line 166, in _restore_aux
    [text[0] + m.mark + text[1]] + text[2:], marks[1:], n)
  File "E:\Python37\lib\site-packages\phonemizer\punctuation.py", line 166, in _restore_aux
    [text[0] + m.mark + text[1]] + text[2:], marks[1:], n)
IndexError: list index out of range

This is after 83e8e62.

instructions for docker users

How does one access one's files from inside the interactive session? That is, if I sudo docker run -it phonemizer /bin/bash I get transported inside a different space, where my files are not available. Right? And this is the only way to call phonemizer in mac?

Minor suggestion: In the instructions for docker users, add the explicit instruction to do git clone https://github.com/bootphon/phonemizer.git.

[OSX] espeak-1.48.03 04.Mar.14: unrecognized option '--sep=_'

On OSX and espeak-1.48.03 04.Mar.14, phonemizer fails with unrecognized option '--sep=_':

$ phonemize -l ca test.txt
espeak: unrecognized option `--sep=_'
espeak: unrecognized option `--sep=_'
espeak: unrecognized option `--sep=_'
espeak: unrecognized option `--sep=_'
espeak: unrecognized option `--sep=_'
espeak: unrecognized option `--sep=_'
espeak: unrecognized option `--sep=_'

Here's what test.txt contains:

corre juga
quin posaras primer
no per quin comences corre
a veure
si
quin hi posaras
quin hi posaras ara

option to choose the espeak binary to use

• from command line, for example phonemize --espeak-path /bin/espeak-ng -b espeak ...
• expose the same option in API as EspeakBackend.__init__(binary=...)

C++

Hey,

Do you know if there is a c++ or equivalent version of this project?

thank you

Support for timing and TextGrid output

Hi, Thank you for this great tool!

I wonder if you plan on supporting other output types like TextGrid that allow synchronization of phoneme and actual time.
https://www.fon.hum.uva.nl/praat/manual/TextGrid_file_formats.html

ImportError: No module named phonemizer.

I am installing facebookresearch/loop, and phonemizer is one of the modules should be imported. However, when I followed the whole installation of phonemizer and told me that finished processing dependencies for phonemizer==0.3.1, I got "ImportError: No module named phonemizer" when I was running the generate.py of voiceloop. I have no idea of why this happen.

Windows issue with NamedTemporaryFile

when i run phonemize on windows 10 , python 3.6 i still have issue. it looks the temp file doesn't created at all. (the backslash issue fixed i can see)

>>> ph=phonemize('Hello World',strip=False,njobs=1,backend='espeak')
Failed to read file 'C:\Users\cinetec\AppData\Local\Temp\tmp5sigu2vf'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\cinetec\AppData\Local\Programs\Python\Python36\lib\site-packages\phonemizer-1.0.1-py3.6.egg\phonemizer\phonemize.py", line 94, in phonemize
    text, separator=separator, strip=strip, njobs=njobs)
  File "C:\Users\cinetec\AppData\Local\Programs\Python\Python36\lib\site-packages\phonemizer-1.0.1-py3.6.egg\phonemizer\backend.py", line 130, in phonemize
    out = self._phonemize_aux(self._list2str(text), separator, strip)
  File "C:\Users\cinetec\AppData\Local\Programs\Python\Python36\lib\site-packages\phonemizer-1.0.1-py3.6.egg\phonemizer\backend.py", line 235, in _phonemize_aux
    shlex.split(command, posix=False)).decode('utf8')
  File "C:\Users\cinetec\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "C:\Users\cinetec\AppData\Local\Programs\Python\Python36\lib\subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['espeak', '-ven-us', '--ipa=3', '-q', '-f', 'C:\\Users\\cinetec\\AppData\\Local\\Temp\\tmp5sigu2vf']' returned non-zero exit status 1.

Phoneme Separator sometimes adds additional separators trailing the end of a word

from phonemizer.separator import Separator
from phonemizer.phonemize import phonemize

_text = 'The lion and the tiger ran.'
_separator = Separator(phone='*')
test_ph = phonemize(_text, separator=_separator, strip=True, njobs=1, backend='espeak', language='en-us')

print(test_ph)

results in

ð*ə l*aɪə*n** æ*n*d ð*ə t*aɪ*ɡ*ɚ ɹ*æ*n

with two separators attached to the end of the phonemized 'lion'.

As opposed to

_separator = phonemizer.separator.Separator(phone='*')
test_ph = phonemize('The lion ran.', separator=_separator, strip=True, njobs=1, backend='espeak', language='en-us')
print(test_ph)

resulting in

ð*ə l*aɪə*n ɹ*æ*n

without trailing separators.

I noticed this around the following samples as well:
the hello but the gives ð*ə h*ə*l*oʊ** b*ʌ*t ð*ə
Here there and everywhere gives h*ɪɹ ð*ɛɹ** æ*n*d ɛ*v*ɹ*ɪ*w*ɛɹ
He was hungry and tired. gives h*iː w*ʌ*z h*ʌ*ŋ*ɡ*ɹ*i** æ*n*d t*aɪɚ*d
He was hungry but tired. gives h*iː w*ʌ*z h*ʌ*ŋ*ɡ*ɹ*i** b*ʌ*t t*aɪɚ*d
The tiger or the lion gives ð*ə t*aɪ*ɡ*ɚ** ɔːɹ ð*ə l*aɪə*n
The lion or the tiger gives ð*ə l*aɪə*n** ɔːɹ ð*ə t*aɪ*ɡ*ɚ

I noticed it around conjunctions like 'and', 'but, and 'or', but not always:
Lions and tigers and bears, oh my! gives l*aɪə*n*z æ*n*d t*aɪ*ɡ*ɚ*z** æ*n*d b*ɛɹ*z oʊ m*aɪ
Lions and tigers run together gives l*aɪə*n*z æ*n*d t*aɪ*ɡ*ɚ*z ɹ*ʌ*n t*ə*ɡ*ɛ*ð*ɚ

logger is None

Hi, it seems that logger can be None in espeak.py. I get the following error:

  File "C:\Users\admin\Anaconda3\envs\TransformerTTS\lib\site-packages\phonemizer\backend\espeak.py", line 290, in _phonemize_aux
    self.logger.warning(
AttributeError: 'NoneType' object has no attribute 'warning'

The method get_logger appears to handle the initialization of a logger or do I have to initialize it manually?

Espeak-ng can be installed from the repos for Debian an Ubuntu (from 18.04)

This should probably be specified in the readme documentation, as it vastly simplifies the install process. I'll do it myself but just put this here as a reminder.

Need more details on Execution

I need to know how to get the phoneme level data by providing the text in us-english like if i give Forget ---- it should give me f' 'er','g','eh','t

Multiprocessing unable to raise warnings

Hi,
when running the phonemization with multiple jobs as in

phonemize(some_text_list,
                             language='de',
                             backend='espeak',
                             strip=False,
                             preserve_punctuation=True,
                             with_stress=False,
                             njobs=16,
                             language_switch='keep-flags')

this error is raised:

---------------------------------------------------------------------------
_RemoteTraceback                          Traceback (most recent call last)
_RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/fcardina/anaconda3/envs/ttsTF/lib/python3.6/site-packages/joblib/externals/loky/process_executor.py", line 418, in _process_worker
    r = call_item()
  File "/Users/fcardina/anaconda3/envs/ttsTF/lib/python3.6/site-packages/joblib/externals/loky/process_executor.py", line 272, in __call__
    return self.fn(*self.args, **self.kwargs)
  File "/Users/fcardina/anaconda3/envs/ttsTF/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 600, in __call__
    return self.func(*args, **kwargs)
  File "/Users/fcardina/anaconda3/envs/ttsTF/lib/python3.6/site-packages/joblib/parallel.py", line 256, in __call__
    for func, args, kwargs in self.items]
  File "/Users/fcardina/anaconda3/envs/ttsTF/lib/python3.6/site-packages/joblib/parallel.py", line 256, in <listcomp>
    for func, args, kwargs in self.items]
  File "/Users/fcardina/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/backend/espeak.py", line 167, in _phonemize_aux
    self._warn_on_lang_switch()
  File "/Users/fcardina/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/backend/espeak.py", line 205, in _warn_on_lang_switch
    self.logger.warning(
AttributeError: 'NoneType' object has no attribute 'warning'
"""

The above exception was the direct cause of the following exception:

AttributeError                            Traceback (most recent call last)
<ipython-input-50-a02c66d832c7> in <module>
      8                              with_stress=False,
      9                              njobs=16,
---> 10                              language_switch='keep-flags')

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/phonemize.py in phonemize(text, language, backend, separator, strip, preserve_punctuation, punctuation_marks, with_stress, language_switch, njobs, logger)
    172     # phonemize the input text
    173     return phonemizer.phonemize(
--> 174         text, separator=separator, strip=strip, njobs=njobs)

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/backend/base.py in phonemize(self, text, separator, strip, njobs)
    116             text = joblib.Parallel(n_jobs=njobs)(
    117                 joblib.delayed(self._phonemize_aux)(t, separator, strip)
--> 118                 for t in chunks(text, njobs))
    119 
    120             # flatten them in a single list

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/joblib/parallel.py in __call__(self, iterable)
   1014 
   1015             with self._backend.retrieval_context():
-> 1016                 self.retrieve()
   1017             # Make sure that we get a last message telling us we are done
   1018             elapsed_time = time.time() - self._start_time

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/joblib/parallel.py in retrieve(self)
    906             try:
    907                 if getattr(self._backend, 'supports_timeout', False):
--> 908                     self._output.extend(job.get(timeout=self.timeout))
    909                 else:
    910                     self._output.extend(job.get())

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/joblib/_parallel_backends.py in wrap_future_result(future, timeout)
    552         AsyncResults.get from multiprocessing."""
    553         try:
--> 554             return future.result(timeout=timeout)
    555         except LokyTimeoutError:
    556             raise TimeoutError()

~/anaconda3/envs/ttsTF/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    430                 raise CancelledError()
    431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
    433             else:
    434                 raise TimeoutError()

~/anaconda3/envs/ttsTF/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

AttributeError: 'NoneType' object has no attribute 'warning'

Setting the number of jobs to 1 does not raise the same error.

Make package pip installable

It would be great to make the package pip installable so that we don't have to clone the repo in order to use the package.

To clarify, I am able to install the package using pip install . from the root of the repo, but cannot find the package in pypi so that I could only install the package using pip install phonemizer.

Looking at the source code, it seems like everything is already in place so that this can be done (according to https://medium.com/@joel.barmettler/how-to-upload-your-python-package-to-pypi-65edc5fe9c56)

add manual stress in a word?

Hello everyone!

Want to pass a sentence in which multiple words have manual stress specified (e.g. "phonem'izer is a perf'ect tool!") where ' denotes stress position. Is it possible?

Thank you!

RuntimeError: espeak not installed on your system

How to solve RuntimeError: espeak not installed on your system on Windows without using Linux in Windows?

Citing phonemizer

Hi,
Is there a citation for phonemizer? Or simply the website.
One of my students is writing up her dissertation and wishes to refer to it :)
Andrew

[espeak-1.48.03] Language switch not happening

Hi,
taken from the documentation, I run

echo "j'aime le football" | phonemize -l fr-fr -b espeak --language-switch keep-flags

and it outputs

ʒɛm ləfutbɔːl

I'm on MacOS catalina, latest phonemizer and espeak versions

Why not word separation for mbrola?

I am trying to use the mbrola extension, thanks for that, but I have troubles with words being concatenated. I this because the time was missing to implement that or is that some sort of mbrola issue?

IPA output for the festival backend

Actually festival uses its own phone set, it would be conveniant to have it's output directly in IPA (as for the espeak backend).

This implies to convert the phones at http://www.festvox.org/bsv/c4711.html to IPA.

Running phonemizer on windows 7 failes

Hi, I'm using phonemizer on windows 7 and python 3.6. Unfortunately, it can not detect my espeak installation. I installed to original binaries and added the installation path to my windows path variable. Furthermore, I installed espeak-ng.

The error is:
RuntimeError: espeak not installed on your system

How to cite this repo?

Hi guys,

thank you for this amazing repo. I am using phonemizer in one of my projects and I would like to give credits in my corresponding paper. How would you like to be cited? Thanks

espeak: option '--ipa' doesn't allow an argument

echo "bonjour le monde" | phonemize -l fr-fr

gives:

espeak: option '--ipa' doesn't allow an argument

speed up phonemizer?

Arabic language support

in this amazing library you are depending on espeak-ng and in thats library support Arabic language culd you please add this feature to your library

installation issues

In mac

No instructions for festival installation in mac OS, instructions on http://www.festvox.org/docs/manual-2.4.0/festival_6.html#Installation not obvious to me.

Installation of espeak:

• download compiled version from http://espeak.sourceforge.net/download.html
• but what next? There is a "Notes-OSX.txt" at the top level and a ReadMe one level down in the downloaded and unzipped file, but neither indicates what the precise next step is (in fact the readme makes reference to a src directory which is not found in the unzipped folder)
• probably due to this, the wordseg tutorial call gives an error:
  Alejandrinas-MacBook-Air:phonemizer acristia$ cat /Users/acristia/Documents/Providence_res/eth01_cds/eth01-ortholines.txt | phonemize --lang en-us -p " " -w ";eword "
  Traceback (most recent call last):
  File "/anaconda/bin/phonemize", line 11, in
  load_entry_point('phonemizer==0.3.1', 'console_scripts', 'phonemize')()
  File "build/bdist.macosx-10.7-x86_64/egg/phonemizer/main.py", line 145, in main
  File "build/bdist.macosx-10.7-x86_64/egg/phonemizer/main.py", line 73, in parse_args
  File "build/bdist.macosx-10.7-x86_64/egg/phonemizer/espeak.py", line 47, in supported_languages
  File "/anaconda/lib/python2.7/subprocess.py", line 212, in check_output
  process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/anaconda/lib/python2.7/subprocess.py", line 390, in init
  errread, errwrite)
  File "/anaconda/lib/python2.7/subprocess.py", line 1024, in _execute_child
  raise child_exception
  OSError: [Errno 2] No such file or directory

On our oberon

[acristia@oberon phonemizer]$ python setup.py install
running install
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

[Errno 13] Permission denied: '/cm/shared/apps/python-anaconda/lib/python2.7/site-packages/test-easy-install-595.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

/cm/shared/apps/python-anaconda/lib/python2.7/site-packages/

Perhaps your account does not have write access to this directory? If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account. If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.

For information on other options, you may wish to consult the
documentation at:

http://packages.python.org/distribute/easy_install.html

Please make the appropriate changes for your system and try again.

[acristia@oberon phonemizer]$ sudo python setup.py install
[sudo] password for acristia:
acristia is not in the sudoers file. This incident will be reported.

Double punctiation break phonemization

I do not have an extensive list, but many double punctuation patterns break the phonemization. One example being !'
Phonemizer from pip version 2.2

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/phonemize.py in phonemize(text, language, backend, separator, strip, preserve_punctuation, punctuation_marks, with_stress, language_switch, njobs, logger)
    172     # phonemize the input text
    173     return phonemizer.phonemize(
--> 174         text, separator=separator, strip=strip, njobs=njobs)

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/backend/espeak.py in phonemize(self, text, separator, strip, njobs)
    233         # finally restore the punctuation
    234         return self._phonemize_postprocess(
--> 235             text, text_type, punctuation_marks)
    236 
    237     def _command(self, fname):

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/backend/base.py in _phonemize_postprocess(self, text, text_type, punctuation_marks)
    138         # restore the punctuation is asked for
    139         if self.preserve_punctuation:
--> 140             text = self._punctuator.restore(text, punctuation_marks)
    141 
    142         # output the result formatted as a string or a list of strings

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/punctuation.py in restore(cls, text, marks)
    147 
    148         """
--> 149         return cls._restore_aux(str2list(text), marks, 0)
    150 
    151     @classmethod

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/punctuation.py in _restore_aux(cls, text, marks, num)
    162             if current.position == 'E':
    163                 return [text[0] + current.mark] + cls._restore_aux(
--> 164                     text[1:], marks[1:], num + 1)
    165             if current.position == 'A':
    166                 return [current.mark] + cls._restore_aux(

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/punctuation.py in _restore_aux(cls, text, marks, num)
    175                 restored = cls._restore_aux(
    176                     [text[0] + current.mark + text[1]] + text[2:],
--> 177                     marks[1:], num)
    178             return restored
    179         else:

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/punctuation.py in _restore_aux(cls, text, marks, num)
    178             return restored
    179         else:
--> 180             return [text[0]] + cls._restore_aux(text[1:], marks, num + 1)

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/punctuation.py in _restore_aux(cls, text, marks, num)
    162             if current.position == 'E':
    163                 return [text[0] + current.mark] + cls._restore_aux(
--> 164                     text[1:], marks[1:], num + 1)
    165             if current.position == 'A':
    166                 return [current.mark] + cls._restore_aux(

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/punctuation.py in _restore_aux(cls, text, marks, num)
    162             if current.position == 'E':
    163                 return [text[0] + current.mark] + cls._restore_aux(
--> 164                     text[1:], marks[1:], num + 1)
    165             if current.position == 'A':
    166                 return [current.mark] + cls._restore_aux(

~/anaconda3/envs/ttsTF/lib/python3.6/site-packages/phonemizer/punctuation.py in _restore_aux(cls, text, marks, num)
    161                     [current.mark + text[0]] + text[1:], marks[1:], num)
    162             if current.position == 'E':
--> 163                 return [text[0] + current.mark] + cls._restore_aux(
    164                     text[1:], marks[1:], num + 1)
    165             if current.position == 'A':

IndexError: list index out of range

Encoding error README in setup

FROM nvidia/cuda:8.0-cudnn7-runtime-ubuntu16.04

VOLUME /data

RUN apt-get update && apt-get install -y python3.5 python3-pip git festival espeak

RUN pip3 install -U pip

#install phonemizer
RUN  git clone https://github.com/bootphon/phonemizer
WORKDIR /phonemizer
RUN python3 setup.py build

Traceback (most recent call last):
  File "setup.py", line 45, in <module>
    long_description=open('README.md').read()
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xca in position 229: ordinal not in range(128)

shared memfd open() failed: Function not implemented

echo "你好世界" | phonemize -l cmn -b espeak --with-stress
shared memfd open() failed: Function not implemented
shared memfd open() failed: Function not implemented
shared memfd open() failed: Function not implemented
shared memfd open() failed: Function not implemented
[ERROR] shared memfd open() failed: Function not implemented
fatal error: espeak failed with return code 0

Is it possible to retrieve the phonamized text with punctuations?

Here is my code to phonemize the text:

text = "Recent research at Harvard has shown meditating for as little as 8 weeks can actually increase, the grey matter in the parts of the brain responsible for emotional regulation and learning!"
seperator = phonemizer.separator.Separator(' |', '', '|')
ph = phonemize(text, separator=seperator, strip=False, njobs=1, backend='espeak', language=language)

previously it'd return comma for punctuations and I'd fix them with a regex but with the new version, punctuations are totally ignored. Is there anyway to keep the punctuations intact.

Festival backend ignores apostrophes

$ echo "I'm looking for a job as a digital assistant." | phonemize
ihm luhkaxng faor ax jhaab aez ax dihjhaxtaxl axsihstaxnt

$ echo "Im looking for a job as a digital assistant." | phonemize
ihm luhkaxng faor ax jhaab aez ax dihjhaxtaxl axsihstaxnt

[OSX] espeak-1.48.03 04.Mar.14: unrecognized option `--sep=_'

On OSX and espeak-1.48.03 04.Mar.14, phonemizer fails with unrecognized option '--sep=_':

$ phonemize -l ca test.txt
espeak: unrecognized option `--sep=_'
espeak: unrecognized option `--sep=_'
espeak: unrecognized option `--sep=_'
espeak: unrecognized option `--sep=_'
espeak: unrecognized option `--sep=_'
espeak: unrecognized option `--sep=_'
espeak: unrecognized option `--sep=_'

Here's what test.txt contains:

corre juga
quin posaras primer
no per quin comences corre
a veure
si
quin hi posaras
quin hi posaras ara

espeak-ng

Hi,
I heard about espeak-ng and am considering installing it: https://github.com/espeak-ng/espeak-ng
Do you know if phonemizer will work with espeak-ng?
regards, Andrew

TypeError: __init__() got an unexpected keyword argument 'use_sampa'

File "/mnt1/phonemize.py", line 109, in phonemize
language, use_sampa=use_sampa, logger=logger)
TypeError: init() got an unexpected keyword argument 'use_sampa'

Don't know what phones belong to which word

I wanted to get pronunciations for a vocab of 200k words, but after calling phonemizer the output is 199999 lines long. So I can't just paste the files, because that would lead to a mismatch.

I don't understand why the output does not have as the first column/field the word ?

Cannot build

$ python setup.py build
Traceback (most recent call last):
File "setup.py", line 20, in <module>
from setuptools import setup, find_packages
ImportError: No module named setuptools

I use ubuntu with preinstalled python version Python 2.7.15rc1

add stress mark to phonemes

Hello everyone!

Just wondering if it's possible to add somehow stress mark to the sentence phonemized?
I mean, not asking backend (e.g. eSpeak) to add that, but not removing it instead.
E.G. (if consider '+' as a stress mark) echo "hello world" | phonemize
$ həloʊ wɜːld
echo "hello+ wo+rld" | phonemize
$ həlo+ʊ wɜː+ld
It would be just awesome, especially for languages like Russian.

espeak-ng version number and phone separator error

Bonjour Mathieu,

A couple of issues after I installed the current version of phonemizer. One is a simple fix, the other is a problem I don't understand:

1, espeak-ng is now version 1.49.3, therefore line 175 in backend.py needs to be updated.

2, I tried to use a space character as phone separator but see the following error:

echo "hello world" | phonemize -l en-us --sep=' '
fatal error: [Errno 2] No such file or directory: '--sep= '

I'm not sure why this happens so at the moment used a hack workaround in line 248, also not specifying a separator in the command line:

w = w.replace('_', ' ')

It's working as I want it to now, so this is not an urgent matter. Thanks for sharing this great resource!
Andrew

multi phone for one word

Hi is it possible that generate multi phones for one word?

[Windows] Program removes backslashes in path when calling subprocess

Traceback (most recent call last):
  File "c:\users\User\appdata\local\programs\python\python36-32\lib\site-packages\phonemizer\festival.py", line 140, in _process
    shlex.split(cmd), stderr=fstderr)
  File "c:\users\User\appdata\local\programs\python\python36-32\lib\subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "c:\users\User\appdata\local\programs\python\python36-32\lib\subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['festival', '-b', 'C:UsersUserAppDataLocalTemptmph1yalspd']' returned non-zero exit status 4294967295.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Programs\Python\Python36-32\Scripts\phonemize-script.py", line 11, in <module>
    load_entry_point('phonemizer==0.3.1', 'console_scripts', 'phonemize')()
  File "c:\users\User\appdata\local\programs\python\python36-32\lib\site-packages\phonemizer\main.py", line 192, in main
    separator=sep, strip=args.strip, njobs=args.njobs, logger=logger)
  File "c:\users\User\appdata\local\programs\python\python36-32\lib\site-packages\phonemizer\phonemize.py", line 113, in phonemize
    separator=separator, strip=strip)
  File "c:\users\User\appdata\local\programs\python\python36-32\lib\site-packages\phonemizer\festival.py", line 77, in phonemize
    b = _process(a, script, logger)
  File "c:\users\User\appdata\local\programs\python\python36-32\lib\site-packages\phonemizer\festival.py", line 149, in _process
    .format(cmd, err.returncode, fstderr.read()))
RuntimeError: Command "festival -b C:\Users\User\AppData\Local\Temp\tmph1yalspd" returned exit status 4294967295, output is:
SIOD ERROR: could not open file C:UsersUserAppDataLocalTemptmph1yalspd

So I think I know what happend:
in the module phonemizer.espeak at line 85 it calls shlex.split(command). when testing, this happend:

>>> command='C:\\Users\\User\\AppData\\Local\\Temp\\test'
>>> shlex.split(command)
['C:UsersUserAppDataLocalTemptest']

This has something to do with Windows paths using backslashes.

This can be proven by replacing the double backslashes with forwardslashes.

Then the path returned keeps its path seperators (forwardslashes).

Edge case where multiline outputs get merged with preserve_punctuations and custom punctuation marks

The following

from phonemizer import phonemize

phonemize(['"Hey! "', '"hey,"'], backend="espeak", preserve_punctuation=True, punctuation_marks='.!;:,?')

Gives for output [' heɪ ! heɪ ', ','], (the second sentence got merged with the first) when [' heɪ ! ', 'heɪ,'] was expected.

RuntimeError: <function name at 0x113970398> not installed on your system

Traceback (most recent call last):
File "generate.py", line 177, in
main()
File "generate.py", line 132, in main
txt = text2phone(args.text, char2code)
File "generate.py", line 52, in text2phone
ph = phonemize(text, separator=seperator)
File "build/bdist.macosx-10.13-x86_64/egg/phonemizer/phonemize.py", line 90, in phonemize
File "build/bdist.macosx-10.13-x86_64/egg/phonemizer/backend.py", line 263, in init
File "build/bdist.macosx-10.13-x86_64/egg/phonemizer/backend.py", line 57, in init
RuntimeError: <function name at 0x113970398> not installed on your system

Issues with diacritics in French

Hello, I found some issues with the French "à" when using the espeak backend. I'm running 2.2.1 and espeak-ng-1.49.2.

Command line:

echo "Je vais à" | phonemize -l=fr-fr -b=espeak
ʒə vɛ

The "à" is simply removed. The same goes for é, I haven't tried other characters.

Python:

>>> phonemize('Je vais à', language="fr-fr", backend="espeak")
'ʒə vɛz aaksɑ̃ɡʁav '

The behaviour isn't consistent with the above, but it is problematic in a different way: here the "à" is explicitly spelled out as "a accent grave"

No module named 'segments'

Hi, I tried to install Phonemizer from sources, when I run "python setup.py", I got the following error:
Traceback (most recent call last):
File "setup.py", line 21, in
import phonemizer
File "/home/duan/phonemizer/phonemizer/init.py", line 17, in
from .phonemize import phonemize
File "/home/duan/phonemizer/phonemizer/phonemize.py", line 27, in
from phonemizer.backend import (
File "/home/duan/phonemizer/phonemizer/backend/init.py", line 19, in
from .segments import SegmentsBackend
File "/home/duan/phonemizer/phonemizer/backend/segments.py", line 20, in
import segments
ModuleNotFoundError: No module named 'segments'

how to convert phonemes to text?

I want to convert sequence of phone like: "tʃ_əː4 n_e1_n t_u6 ɗ_o6_ŋ" back to text (sequence of characters): "trở nên thụ động"
thanks <3

inconsistant punctuation processing

When a point is separating two sentences (in a single line of text), the output gives a single utterance. But when replacing the point by a comma or a semicolon, this outputs 2 utterances. See below:

$ echo 'a comma a point.' | phonemize
ɐ kɑːmə ɐ pɔɪnt 
$ echo 'a comma. a point.' | phonemize
ɐ kɑːmə ɐ pɔɪnt 
$ echo 'a comma; a point.' | phonemize
ɐ kɑːmə 
ɐ pɔɪnt 
$ echo 'a comma, a point.' | phonemize
ɐ kɑːmə 
ɐ pɔɪnt 
$ echo 'a comma? a point!' | phonemize
ɐ kɑːmə 
ɐ pɔɪnt 

The expected behavior would be to ignore punctuation.

Bug in processing some characters.

The phonemizer 2.1 raises error when I call it the following sentence

'Truly replied, "Yes."'

  File "/home/erogol/miniconda3/lib/python3.7/site-packages/phonemizer-2.1-py3.7.egg/phonemizer/punctuation.py", line 166, in _restore_aux
    [text[0] + m.mark + text[1]] + text[2:], marks[1:], n)
IndexError: list index out of range

Phonemizer gettiing stuck for some texts

Filipino (Tagalog) language support?

Support for more languages in the festival backend

Add at least British English, Spanish and Welsh, which are distributed with festival. Actually only American English is supported.