System.FormatException: Input string was not in a correct format

[DaveSkender]

System.FormatException: Input string was not in a correct format

[LeeDongGeon1996]

After quick search, I found that we might convert Decimal into str incorrectly. Looks like just using str() is not affected by locale setting.
See here: https://stackoverflow.com/questions/31910741/how-can-i-locale-format-a-python-decimal-and-preserve-its-precision

[ArtemHachaturov]

There is a small advance, the values fall into quotes

import pandas as pd
from stock_indicators import indicators, Quote
from babel.numbers import format_decimal

df = pd.read_csv('quotes.csv', parse_dates=['Time'])

def format_with_babel(number, locale='ru_RU'):
return format_decimal(number, locale=locale)

df['Open'] = df['Open'].apply(format_with_babel)
df['High'] = df['High'].apply(format_with_babel)
df['Low'] = df['Low'].apply(format_with_babel)
df['Close'] = df['Close'].apply(format_with_babel)
df['Volume'] = df['Volume'].apply(lambda x: format_with_babel(x, locale='ru_RU'))

quotes = [
Quote(date, open, high, low, close, volume)
for date, open, high, low, close, volume
in zip(df['Time'], df['Open'], df['High'], df['Low'], df['Close'], df['Volume'])
]
print(quotes)

#[<stock_indicators.indicators.common.quote.Quote object at 0x0000023D856F24C0>, <stock_indicators.indicators.common.quote.Quote object at 0x0000023D88E87080>....

print("Time | Open | High | Low | Close | Volume")
for q in quotes:
print(f"{q.date:%Y-%m-%d} | {q.open} | {q.high} | {q.low} | {q.close} | {q.volume})")

#But at the end there is an error

Time | Open | High | Low | Close | Volume
Traceback (most recent call last):
File "e:\Binance Bot\python\12.py", line 34, in
print(f"{q.date:%Y-%m-%d} | {q.open} | {q.high} | {q.low} | {q.close} | {q.volume})")
^^^^^^
File "C:\Users\russt\AppData\Local\Programs\Python\Python311\Lib\site-packages\stock_indicators\indicators\common\quote.py", line 22, in _get_open
return to_pydecimal(quote.Open)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\russt\AppData\Local\Programs\Python\Python311\Lib\site-packages\stock_indicators_cstypes\decimal.py", line 36, in to_pydecimal
return PyDecimal(str(cs_decimal))
^^^^^^^^^^^^^^^^^^^^^^^^^^
decimal.InvalidOperation: [<class 'decimal.ConversionSyntax'>]

[DaveSkender]

I think we're going to need to setup a rather elaborate set of tests using different locale settings on a GitHub Actions runner to better understand exactly what's happening. It's difficult to test on a local computer, and I'm not 100% confident I know what's happening. ru-Ru is a good test case to start with.

Any one of these can be a problem if they're not aligned on locale for both date/time and numerics:

• source quotes format
• environment/machine locale
• dotnet native locale
• python native locale
• Python.NET locale handling
• pandas dataFrame locale handling
• DLL compiled default locale handling

cc @LeeDongGeon1996

[ArtemHachaturov]

Hi, @ArtemHachaturov . Thanks for your detail report. As I mentioned above, it looks like the locale setting does not apply in converting into str Could you give me the results from the code below? That would be very helpful to find the reason

from decimal import Decimal
value = Decimal(12.345)
print(value) # case 1
print('{:n}'.format(value)) # case 2
print(format(value, 'n')) # case 3

from stock_indicators._cslib import CsDecimal
print(CsDecimal.Parse(format(value, 'n'))) # case 4

Thank you very much to you and your team for your feedback and for your responsiveness in solving this problem, I really appreciate it.

Here is the output of this code

12.3450000000000006394884621840901672840118408203125
12.3450000000000006394884621840901672840118408203125
12.3450000000000006394884621840901672840118408203125
Traceback (most recent call last):
File "e:\Binance Bot\python\14.py", line 8, in
print(CsDecimal.Parse(format(value, 'n'))) # case 4
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
System.FormatException: Input string was not in a correct format. at System.Number.ThrowOverflowOrFormatException(ParsingStatus
status, TypeCode type)
at System.Decimal.Parse(String s)

---

12.3450000000000006394884621840901672840118408203125
12.3450000000000006394884621840901672840118408203125
12.3450000000000006394884621840901672840118408203125
Traceback (most recent call last):
File "e:\Binance Bot\python\14.py", line 8, in
print(CsDecimal.Parse(format(value, 'n'))) # case 4
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
System.FormatException: The input string '12.3450000000000006394884621840901672840118408203125' was not in a correct format.
at System.Number.ThrowFormatException[TChar](ReadOnlySpan1 value) at System.Number.ParseDecimal[TChar](ReadOnlySpan1 value, NumberStyles styles,
NumberFormatInfo info)
at System.Decimal.Parse(String s)
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj,
Span`1 copyOfArgs, BindingFlags invokeAttr)

[DaveSkender]

I suspect this essentially reset your PC environment locale to en_US, which would certainly solve your blocking issue; however, we’re still going to try and get it working better on other locales.

[SnowyOw1]

@SnowyOw1 Thank you for reporting. We are fixing it now here. Could you try with this patched version? (it's still in reviewing yet but about to be merged.)

+and If you let me know your locale setting on your machine, it'll be very appreciated.

I hereby confirrm the fix is working. I didn't apply the full patch but monkey-patched sources.

My locale is

PS C:\> Get-WinSystemLocale

LCID             Name             DisplayName
----             ----             -----------
1049             ru-RU            Russian (Russia)


PS C:\> (Get-WinSystemLocale).NumberFormat


CurrencyDecimalDigits    : 2
CurrencyDecimalSeparator : ,
IsReadOnly               : False
CurrencyGroupSizes       : {3}
NumberGroupSizes         : {3}
PercentGroupSizes        : {3}
CurrencyGroupSeparator   :  
CurrencySymbol           : ₽
NaNSymbol                : не число
CurrencyNegativePattern  : 8
NumberNegativePattern    : 1
PercentPositivePattern   : 1
PercentNegativePattern   : 1
NegativeInfinitySymbol   : -∞
NegativeSign             : -
NumberDecimalDigits      : 2
NumberDecimalSeparator   : ,
NumberGroupSeparator     :  
CurrencyPositivePattern  : 3
PositiveInfinitySymbol   : ∞
PositiveSign             : +
PercentDecimalDigits     : 2
PercentDecimalSeparator  : ,
PercentGroupSeparator    :  
PercentSymbol            : %
PerMilleSymbol           : ‰
NativeDigits             : {0, 1, 2, 3...}
DigitSubstitution        : None

[DaveSkender]

There's also this <NeutralLanguage>en-US</NeutralLanguage> tag hard-coded into the .NET library, but unsure if it's a problem.

[ArtemHachaturov]

Yes, I agree with you, of course, but I just want to solve my problem

[DaveSkender]

Yes, I agree with you, of course, but I just want to solve my problem

If locale were compatible in all of the components listed above in your environment it should work. Until I run some more tests with all possible scenarios, I really can't tell which compatibility scenarios are the most problematic ... I honestly don't know where your implementation is failing.

It works with all the default settings that the GitHub Actions use, which is likely en_US.UTF-8.

[LeeDongGeon1996]

Yeah we need to have some tests for this. I'll have a look when I get back home (I'm in workshop now)

[LeeDongGeon1996]

Hi, @ArtemHachaturov . Thanks for your detail report. As I mentioned above, it looks like the locale setting does not apply in converting into str
Could you give me the results from the code below? That would be very helpful to find the reason

from decimal import Decimal
value = Decimal(12.345)
print(value) # case 1
print('{:n}'.format(value)) # case 2
print(format(value, 'n')) # case 3

from stock_indicators._cslib import CsDecimal
print(CsDecimal.Parse(format(value, 'n'))) # case 4

[LeeDongGeon1996]

@ArtemHachaturov
Ohhhh! Thank you. That was very helpful to figure out what's the problem.
Could you give me more information? It looks like your python runtime is using the locale which uses period(.) as a decimal point while your DotNet runtime is using comma(,) as a decimal point.
If you can give me the results after running the below, It'll be very appreciated.

On Python:

import locale
print(locale.getlocale(locale.LC_ALL))
print(locale.localeconv())

On your terminal:

locale

[ArtemHachaturov]

Dear friends, I am glad to inform you that everything worked out for me and now everything works like a clock.

I had only one locale, this was RU, I understood that I still needed to install the language pack us, but for certain reasons I was unable to do this until I completely demolished the system and installed another Windows, where I also tried to download it manually the language pack was installed but was not applied until I used the following commands

• cmd
  control intl.cpl,, /f:"%SystemRoot%\System32\LangRes.dll",/s:"0x00000409"

-PowerShell
Set-WinSystemLocale en-US
Set-Culture en-US

and restarting the computer

Then I used check current locale

import locale

current_locale = locale.getdefaultlocale()[0]

print(current_locale) #en_US

I realized that everything should work!

Everything looks amazing

df = pd.read_csv('quotes.csv', parse_dates=['Time'])
quotes = [
Quote(date, open, high, low, close, volume)
for date, open, high, low, close, volume
in zip(df['Time'], df['Open'], df['High'], df['Low'], df['Close'], df['Volume'])
]

print("Time | Open | High | Low | Close | Volume")
for q in quotes:
print(f"{q.date:%Y-%m-%d} | {q.open} | {q.high} | {q.low} | {q.close} | {q.volume})")

2018-07-27 | 275.57 | 275.68 | 272.34 | 273.35 | 79050080)
2018-07-30 | 273.44 | 273.61 | 271.35 | 271.92 | 65624404)
2018-07-31 | 272.76 | 273.93 | 272.34 | 273.26 | 70594928)
2018-08-01 | 273.49 | 274.04 | 272.1 | 272.81 | 55443260)
2018-08-02 | 271.38 | 274.48 | 271.15 | 274.29 | 65298924)
2018-08-03 | 274.43 | 275.52 | 274.23 | 275.47 | 55527740)
2018-08-06 | 275.51 | 276.82 | 275.08 | 276.48 | 40564136)
2018-08-07 | 277.21 | 277.81 | 277.06 | 277.39 | 44471960)

Thank you very much to you and your team, I wish you great success and prosperity

[SnowyOw1]

I am able to reproduce the symptom. Resetting system locale is clearly not an option in my case.

Indeed the problem is at
CsDecimal.Parse(str(decimal))

import locale
print(locale.localeconv())
print(locale.getlocale(locale.LC_ALL))

{'int_curr_symbol':` '', 'currency_symbol': '', 'mon_decimal_point': '', 'mon_thousands_sep': '', 'mon_grouping': [], 'positive_sign': '', 'negative_sign': '', 'int_frac_digits': 127, 'frac_digits': 127, 'p_cs_precedes': 127, 'p_sep_by_space': 127, 'n_cs_precedes': 127, 'n_sep_by_space': 127, 'p_sign_posn': 127, 'n_sign_posn': 127, 'decimal_point': '.', 'thousands_sep': '', 'grouping': []}

File "", line 1, in
File "C:\Program Files\Python311\Lib\locale.py", line 608, in getlocale
raise TypeError('category LC_ALL is not supported')

[LeeDongGeon1996]

@SnowyOw1 Thank you for reporting. We are fixing it now here. Could you try with this patched version? (it's still in reviewing yet but about to be merged.)

pip uninstall stock-indicators
pip install git+https://github.com/LeeDongGeon1996/Stock.Indicators.Python.git@fix/apply-locale

+and If you let me know your locale setting on your machine, it'll be very appreciated.

[LeeDongGeon1996]

Patched version released on test PyPI. Please have a test and any feedback appreciated.

pip install -i https://test.pypi.org/simple/ stock-indicators==1.3.1.dev9

[DaveSkender]

Thank you for confirming. This patch has been published in v1.3.1