Chinese symbols in DHCP lease

[luqasz]

Chinese symbols in DHCP lease

[luqasz]

I have no idea how to fix this issue.

[luqasz]

How can I pass the params like that {'dynamic': False} in any print?

You have to use api query. I plan to add support for it in #11 but it is more tricky then I thought.

from your apicli.py everything was working fine

What you did exactly so that decoding works in apicli.py ?

[lst123]

Sorry for misleading you. I've just put these symbols into REPL and do .decode(decode('utf-16'))

[luqasz]

Have you tried changing encoding ?

api = connect(username='admin', password='abc', host='some.address.com', encoding='UTF-16')

[lst123]

Yes I've tried. With these code:
routeros = connect(username=default_user,
password=default_secret, host=lan_ip, timeout=90, encoding='UTF-16')

I had: exc.ConnectionError: Socket timed out. timed out
When I change to UTF-8, my connection is OK.

[luqasz]

In [12]: '/login'.encode('UTF-8')
Out[12]: b'/login'

In [13]: '/login'.encode('UTF-16')
Out[13]: b'\xff\xfe/\x00l\x00o\x00g\x00i\x00n\x00'

This is why you can not use utf-16 and you get a timeout. There is nothing I can do since API does not recognize utf-16.

Scratch that one. I think I can change the way encoding / decoding works. Later I will write some more info.

[luqasz]

I will try with encoding just a value of a word e.g. =host-name=ENCODED_VALUE

[luqasz]

@lst123 I've pushed value-encoding branch that works as described above. Pass utf-16 encoding and check if you see Chinese symbols. Let me know.

[lst123]

@luqasz I've tried your changes and got this error:
(virtualenv) pc@npc-PC:/git/value_encoding/virtualenv$ ./utf16.py
'ascii' codec can't encode characters in position 0-15: ordinal not in range(128)
(virtualenv) pc@npc-PC:/git/value_encoding/virtualenv$

My connection string was:
s = connect(host=lan_ip, username=default_user,
password=default_pw, timeout=120, encoding='UTF-16')
I executed these code:
dhcp_l = '/ip/dhcp-server/lease/print'
d_l = s(cmd=dhcp_l)
print(d_l)

[luqasz]

Try apicli.py. It will print out output and crash when can not decode. Please paste output here.

[lst123]

@luqasz hello. Please see the output:
(virtualenv) pc@npc-PC:/git/value_encoding$ ./apicli.py 10.6.68.1 -u testuser
Password:
<--- '/login'
<--- '=name=dummy_user'
<--- '=password=dummy_password'
<--- EOS
---> '!done'
---> '=ret=㐲户㕦㈴㐵攳㌹挷つ㙤㐷ㄲ昱㔵㠵㔶'
---> EOS
Traceback (most recent call last):
File "./apicli.py", line 75, in
main()
File "./apicli.py", line 58, in main
api = connect(args.host, args.user, pw, logger=mainlog, encoding='UTF-16')
File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/init.py", line 58, in connect
api('/login', **{'name': username, 'response': encode_password(token, password)})
File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/init.py", line 75, in encode_password
token = token.encode('ascii', 'strict')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128)
(virtualenv) pc@npc-PC:~/git/value_encoding$

[luqasz]

You can not connect because you have Chinese symbols in password. Am I right ? Please try ASCII only characters in password and then login and print e.g. DHCP leases which have as far as I remember Chinese symbols.

[lst123]

Hello.
If I open a connection with apicli.py:

api = connect(args.host, args.user, pw, logger=mainlog, encoding='UTF-16')

with a user: testuser and password: testpassword there will be an error:
(virtualenv) pc@npc-PC:/git/value_encoding/virtualenv$ ./apicli.py 10.6.68.1 -u testuser
Password:
<--- '/login'
<--- '=password=dummy_password'
<--- '=name=dummy_user'
<--- EOS
---> '!done'
---> '=ret=〷㈰㝣〱搱㈶㉦㘸㈲㘳戵ㄱ㜴ㄷ戱㍤'
---> EOS
Traceback (most recent call last):
File "./apicli.py", line 75, in
main()
File "./apicli.py", line 58, in main
api = connect(args.host, args.user, pw, logger=mainlog, encoding='UTF-16')
File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/init.py", line 58, in connect
api('/login', **{'name': username, 'response': encode_password(token, password)})
File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/init.py", line 75, in encode_password
token = token.encode('ascii', 'strict')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128)
(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$

If I remove an encoding statement, the script will suffer from chinese symbols:
Traceback (most recent call last):
File "./apicli.py", line 75, in
main()
File "./apicli.py", line 65, in main
selectloop(api)
File "./apicli.py", line 42, in selectloop
proto.readSentence()
File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/connections.py", line 147, in readSentence
sentence = tuple(word for word in iter(self.readWord, b'\x00'))
File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/connections.py", line 147, in
sentence = tuple(word for word in iter(self.readWord, b'\x00'))
File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/connections.py", line 168, in readWord
value = value.decode(encoding=self.encoding, errors='strict')
UnicodeDecodeError: 'ascii' codec can't decode byte 0x8f in position 3: ordinal not in range(128)

[luqasz]

<--- '/login'
<--- '=password=dummy_password'
<--- '=name=dummy_user'
<--- EOS
---> '!done'
---> '=ret=〷㈰㝣〱搱㈶㉦㘸㈲㘳戵ㄱ㜴ㄷ戱㍤'
---> EOS

This tells me that you still have Chinese symbols in password. Even though you've stated that you have testpassword set.

If I remove an encoding statement, the script will suffer from chinese symbols:

What encoding statement ?

Be specific and exact.
Use markdown for code samples.

[lst123]

Hello.

I don't understand why you think that I've used chinese symbols in the password.
In order to prove you my statements I've changed your apicli.py a little bit:

(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ diff apicli.py new_apicli.py 
22a23,24
> argParser.add_argument(
>         '-pw', '--password', type=str, help="password")
56c58
<     pw = getpass.getpass()
---
>     #pw = getpass.getpass()
58c60
<         api = connect(args.host, args.user, pw, logger=mainlog)
---
>         api = connect(args.host, args.user, args.password, logger=mainlog, encoding='UTF-16')
(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ 

When I try to open a connection with this credentials:

(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ ./new_apicli.py 10.6.68.1 -u testuser --password testuser
<--- '/login'
<--- '=password=dummy_password'
<--- '=name=dummy_user'
<--- EOS
---> '!done'
---> '=ret=摣㙥㔳㌲㥦攲㜶㉥㘹昴捦昲㙣㝣晤换'
---> EOS
Traceback (most recent call last):
  File "./new_apicli.py", line 77, in <module>
    main()
  File "./new_apicli.py", line 60, in main
    api = connect(args.host, args.user, args.password, logger=mainlog, encoding='UTF-16')
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/__init__.py", line 58, in connect
    api('/login', **{'name': username, 'response': encode_password(token, password)})
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/__init__.py", line 75, in encode_password
    token = token.encode('ascii', 'strict')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128)

I get this error.
Even if I put some credentials that don't exist on my Mikrotik router, I'll get the same error (but "=ret" line, differs a little bit):

(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ ./new_apicli.py 10.6.68.1 -u mrpickles --password mrpickles
<--- '/login'
<--- '=password=dummy_password'
<--- '=name=dummy_user'
<--- EOS
---> '!done'
---> '=ret=挳搸っ㉥愴捤㔵挷换ㅥ㐱㐲昶㡢〸㍢'
---> EOS
Traceback (most recent call last):
  File "./new_apicli.py", line 77, in <module>
    main()
  File "./new_apicli.py", line 60, in main
    api = connect(args.host, args.user, args.password, logger=mainlog, encoding='UTF-16')
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/__init__.py", line 58, in connect
    api('/login', **{'name': username, 'response': encode_password(token, password)})
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/__init__.py", line 75, in encode_password
    token = token.encode('ascii', 'strict')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128)

After that I've changed encoding to UTF-8:

(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ diff apicli.py new_apicli.py 
22a23,24
> argParser.add_argument(
>         '-pw', '--password', type=str, help="password")
56c58
<     pw = getpass.getpass()
---
>     #pw = getpass.getpass()
58c60
<         api = connect(args.host, args.user, pw, logger=mainlog)
---
>         api = connect(args.host, args.user, args.password, logger=mainlog, encoding='UTF-8')
(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ 

Then I test it:

(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ ./new_apicli.py 10.6.68.1 -u testuser --password testuser
<--- '/login'
<--- '=name=dummy_user'
<--- '=password=dummy_password'
<--- EOS
---> '!done'
---> '=ret=d7ba7817dcb7ed0870f8fdaf4b3e8040'
---> EOS
<--- '/login'
<--- '=name=testuser'
<--- '=response=006ef45bda167b800723f613966f93dfa9'
<--- EOS
---> '!done'
---> EOS
/ip/dhcp-server/lease/print

<--- '/ip/dhcp-server/lease/print'
<--- EOS
---> '!re'
---> '=.id=*4'
---> '=address=10.6.68.250'
---> '=mac-address=4C:F9:2C:A0:E3:B0'
---> '=address-lists='
---> '=server=dhcp1'
---> '=dhcp-option='
---> '=status=bound'
---> '=expires-after=17h18m32s'
---> '=last-seen=6h41m28s'
---> '=active-address=10.6.68.250'
---> '=active-mac-address=4C:F9:2C:A0:E3:B0'
---> '=active-server=dhcp1'
---> '=host-name=NPIA0E3B0'
---> '=radius=false'
---> '=dynamic=false'
---> '=blocked=false'
---> '=disabled=false'
---> EOS
---> '!re'
---> '=.id=*7DC'
---> '=address=10.6.68.234'
---> '=mac-address=7D:34:D3:E3:B3:A7'
---> '=client-id=1:7D:34:d3:e3:b3:a7'
---> '=address-lists='
---> '=server=dhcp1'
---> '=dhcp-option='
---> '=status=bound'
---> '=expires-after=14h27m43s'
---> '=last-seen=2m2s'
---> '=active-address=10.6.68.234'
---> '=active-mac-address=7D:34:D3:E3:B3:A7'
---> '=active-client-id=1:7D:34:d3:e3:b3:a7'
---> '=active-server=dhcp1'
---> '=host-name=wbf-342'
---> '=radius=false'
---> '=dynamic=true'
---> '=blocked=false'
---> '=disabled=false'
---> EOS
Traceback (most recent call last):
  File "./new_apicli.py", line 77, in <module>
    main()
  File "./new_apicli.py", line 67, in main
    selectloop(api)
  File "./new_apicli.py", line 44, in selectloop
    proto.readSentence()
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/connections.py", line 147, in readSentence
    sentence = tuple(word for word in iter(self.readWord, b'\x00'))
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/connections.py", line 147, in <genexpr>
    sentence = tuple(word for word in iter(self.readWord, b'\x00'))
  File "/home/pc/git/value_encoding/virtualenv/lib/python3.5/site-packages/librouteros/connections.py", line 168, in readWord
    value = value.decode(encoding=self.encoding, errors='strict')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8f in position 3: invalid start byte
(virtualenv) pc@npc-PC:~/git/value_encoding/virtualenv$ 

I can send a TCPdump file to you, if you don't believe me after all.

[luqasz]

Ok. Value of =ret= is encoded by api with UTF-16 which results in some symbols:

In [4]: b'd7ba7817dcb7ed0870f8fdaf4b3e8040'.decode('UTF-16')
Out[4]: '㝤慢㠷㜱捤㝢摥㠰〷㡦摦晡戴攳〸〴'

As for other values and utf16, it is impossible to implement thath. Observe:

In [6]: '7D:34:D3:E3:B3:A7'.encode('UTF16')
Out[6]: b'\xff\xfe7\x00D\x00:\x003\x004\x00:\x00D\x003\x00:\x00E\x003\x00:\x00B\x003\x00:\x00A\x007\x00'

In [7]: '7D:34:D3:E3:B3:A7'.encode('UTF-16')
Out[7]: b'\xff\xfe7\x00D\x00:\x003\x004\x00:\x00D\x003\x00:\x00E\x003\x00:\x00B\x003\x00:\x00A\x007\x00'

In [8]: '7D:34:D3:E3:B3:A7'.encode('UTF-8')
Out[8]: b'7D:34:D3:E3:B3:A7'

In [9]: b'7D:34:D3:E3:B3:A7'.decode('utf16')
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-9-560726224de8> in <module>()
----> 1 b'7D:34:D3:E3:B3:A7'.decode('utf16')

UnicodeDecodeError: 'utf-16-le' codec can't decode byte 0x37 in position 16: truncated data

Can you read correct Chinese symbols in winbox, ssh terminal ? MikroTik does not support any kind of encoding. In winbox it uses encoding which is set in your windows machine. MikroTik just stores bytes.

[lst123]

Thank you for your reply.

Can you read correct Chinese symbols in winbox, ssh terminal?

No, I can't.

[feute]

I've come across with this issue when I was trying to make a script to take RouterOS users from the Hotspot and save them to the database.

In my case, the error message was:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 17: invalid continuation byte

I tried to do this with Go, and it read it properly in some cases (it won't print the character on the terminal, but it did showed correctly in a file). But when trying to insert to the database, it showed the same error about the \xf1. I figured this is shown in Winbox as in ñ; I searched in Google the hex code for that character and it showed a different one, not 0xf1, but 0xc3b1. I replaced any occurrences of the invalid byte f1 to the literal letter, and it worked.

I think Winbox has a problem with encoding characters, showing these invalid bytes; it truncates (or corrupts) the characters. So, when trying to decode it, it shows this error.

I couldn't manage to fix this with librouteros since I can't touch the retrieved data, it throws the exception before doing anything with it. But it might give you an idea of why this happens and how to (possibly) fix it.

[luqasz]

Hi.
Thx for information. How MikroTik stores and then returns non ASCII characters is really a mystery. I do not understand how it is done. I also haven't seen code that reliably handles those use cases.