Error importing encrypted and plain Aegis json files.

[marcopaganini]

Error importing encrypted and plain Aegis json files.

[marcopaganini]

Sure. I'm sending a single account snippet that causes the problem. I've changed the UUID and the key for privacy purposes. Unfortunately, github does not want to accept a json or even a json.gz file, so I'm sending it inline below:

{
    "version": 1,
    "header": {
        "slots": null,
        "params": null
    },
    "db": {
        "version": 2,
        "entries": [
            {
                "type": "totp",
                "uuid": "f64dbf30-2156-4df2-951d-801dfc1fffff",
                "name": "[email protected]",
                "issuer": "PayPal",
                "info": {
                    "secret": "RP3ULJWYHJGO7UHB",
                    "algo": "SHA1",
                    "digits": 6,
                    "period": 30
                }
            }
        ]
    }
}

[marcopaganini]

That is fantastic news Paolo! Thanks for persevering in this bug report!

[paolostivanin]

Hello,
I've just tried with the same versions, and everything works fine. If you could share a reproducer, that would help a lot with debugging. You can change the secret, that one I don't care, but please keep the account and issuer the same. I suspect this has something to do with utf8.

[paolostivanin]

I've found a fix for the plain issue (I'm using the old aegis's format, v1, while now it's v2), but the encrypted one puzzles me (because importing my encrypted data works fine as of yesterday).
Can you share an encrypted version that's failing?

Thanks

[marcopaganini]

Thanks! So I'm having some problems generating an encrypted file. If I do, it will contain all of my credentials, before I have a chance to edit it and change the keys. I tried adding a new user to my phone (so I can install Aegis again, import one account, and export an encrypted json) but unfortunately my device is company owned and won't let me add new users.

Any ideas?

[paolostivanin]

One thing you could do is to decrypt your file using the upstream tool, then check if it is valid json using, for example, python.
If everything checks out fine, then you could try to compile this branch and execute otpclient from the terminal (./otpclient).
Then please paste the output here!

Thanks

[marcopaganini]

I've compiled otpclient from branch-281. Here are my observations:

Attempt 1: Encrypted export from Aegis (FAIL)

• Attempted to import my original (aegis generated) encrypted file.
• After entering my password, get the "An error occurred while importing, no nothing has been added to the database. The error is: end of file expected near '@'.
• stdout messages:

before header
after header
after derive
before open cipher and set data
after check tag
before parse_json_data
error while loading json
after parse_json_data

Attempt 2: Re-encrypted export from otpclient debug-281 (FAIL)

• Decrypted my aegis encrypted import using upstream.
• Imported the plain result into otpclient debug-281
• Exported to an encrypted file (~/aegis_encrypted.json)
• Removed the database
• Restarted otpclient
• Attempted to import the encrypted ~/.aegis_encrypted.json file.
• After entering my password, get the "An error occurred while importing, no nothing has been added to the database. The error is: end of file expected near '@'.
• stdout messages:

before header
after header
after derive
before open cipher and set data
after check tag
before parse_json_data
error while loading json
after parse_json_data

Note: All json files are parsed by jq without any problems.

Does this help in any way?

[paolostivanin]

It helped confirming where the bug is happening (json_loads) but, unfortunately, it doesn't really help me understanding why.
Especially with the 2 attempt: you were able to import the plain JSON and exporting it, but you were not able to load it again later on 😵
Do me a favor: from the decrypted JSON, clean up all entries but one, import it, export it encrypted and try to import it again. Does it fail?
If yes, please share the JSON with me.

Thanks!

[paolostivanin]

Also, I've just updated the branch debug-281. Could you please try to import your db again?
This time you should see it printed on stdout. Is it the same as the plain you have or are there any differences?

[marcopaganini]

Also, I've just updated the branch debug-281. Could you please try to import your db again? This time you should see it printed on stdout. Is it the same as the plain you have or are there any differences?

Just did that and the resulting json output on stdout is identical to the plain version of my encrypted json file. Diff shows no differing lines. The import still failed.

Do me a favor: from the decrypted JSON, clean up all entries but one, import it, export it encrypted and try to import it again. Does it fail? If yes, please share the JSON with me.

Hmm, that's interesting.

Just copied one entry from the full plain json file from aegis into another file and imported it. Re-exported as encrypted and was able to load it without any problems.

Re-exporting the entire file does not seem to work. The funny thing is that I tried bisecting the file manually (26 entries). If I drop the first 13 in otpclient and save the last 13 as encrypted json, it reimports without problems. But then, to my surprise, if I delete the last 13 entries and only export the first 13 ones as encrypted, it also loads (?!)

[marcopaganini]

So here's another interesting data point: If I just load the plain aegis json database on otpclient and save it to disk encrypted, I cannot re-load that encrypted file. However, if I load the plain aegis json database, change any row (I just changed the name in one row) and export an encrypted file, it appears I can read that file.

[paolostivanin]

😮 😮 😮 now I must say I'm really puzzled. I tried multiple things on my side (import/export), but I was always able to successfully complete those actions.
I really need a reproducer, otherwise I'm not gonna be able to help 😞 if you find a way to share with me an encrypted json, please do so.

[paolostivanin]

@marcopaganini let's try on last time. Please checkout the latest changes I made on debug-281 and try to import the full encrypted json.
Then paste here the the results, including the json content from the error (it will tell you line, column, position)

Thanks

[marcopaganini]

Trying to import the encrypted json file generated by OTPClient itself, I get:

error while loading json
======================
end of file expected near '@'
------------
<string>
------------
error on line: 1
------------
error on col: 4577
------------
position: 4577
======================

[marcopaganini]

So is this error relative to the encrypted file or the json post decryption? Looking at the encrypted file, I see a long line (~6.5k) and position 4577 doesn't show anything particularly interesting. Also, the "end of file expected near '@'" message is puzzling (there's not a single "@" sign in this entire file).

[paolostivanin]

The errors is about the decrypted JSON.
It could mean that there is an UTF-8 char or some symbol that's not correctly parsed .

[marcopaganini]

Ha, that's interesting!

In the decrypted file (with upstream otp-decrypt), line 1 has 4576 characters, and the JSON complains about position 4577 (!?) I also took a few minutes to port upstream otp-decrypt to Go (where I can even generate the OTPs myself) and the results match.

Puzzling!

[paolostivanin]

I really need a reproducer to debug, because I'm out of ideas!
Please ping me if you'll ever come up with one, thanks!!

[marcopaganini]

I'll try to backup my aegis configuration and remove all accounts until only one is left. I'll then try to reproduce the bug and send you the file.

…

On Sat, Feb 4, 2023, 01:58 Paolo Stivanin ***@***.***> wrote: I really need a reproducer to debug, because I'm out of ideas! Please ping me if you'll ever come up with one, thanks!! — Reply to this email directly, view it on GitHub <#281 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYUYBAL36UBI354UZI5SE3WVYR3PANCNFSM6AAAAAAUKDZVZM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[marcopaganini]

So, good news. It looks like the approach above worked! I exported my entire vault on Aegis and removed all but three entries. I then changed the password to "password" and exported those into an encrypted json file. OTPclient cannot import that file with the same error as before.

The file is encrypted with "password" as the password. None of the accounts there are valid, so no risk involved.

Hope it helps.
aegis-export-20230204-221858-3password.tar.gz

[paolostivanin]

Cool, that's an amazing news!!
Thanks a lot, I'll get to it ASAP 😊

[paolostivanin]

Wanna laugh? works fine here 😮
What OS are you using?

[marcopaganini]

Holy guacamole!!! 😆 So it has to be in the environment!

I'm using Debian bookworm.

I'm sending the output of ldd otpclient. It could be interesting to compare with yours to see which versions differ (naturally, the output will need to be massaged some to make it work):

Long output follows...

	linux-vdso.so.1 (0x00007ffd1f706000)
	libgtk-3.so.0 => /lib/x86_64-linux-gnu/libgtk-3.so.0 (0x00007fddbc052000)
	libgdk_pixbuf-2.0.so.0 => /lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0 (0x00007fddbc022000)
	libgio-2.0.so.0 => /lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x00007fddbbe42000)
	libgobject-2.0.so.0 => /lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007fddbbde2000)
	libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007fddbbcaa000)
	libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007fddbbb62000)
	libcotp.so.12 => /lib/x86_64-linux-gnu/libcotp.so.12 (0x00007fddbbb5a000)
	libbaseencode.so.1 => /lib/x86_64-linux-gnu/libbaseencode.so.1 (0x00007fddbbb52000)
	libpng16.so.16 => /lib/x86_64-linux-gnu/libpng16.so.16 (0x00007fddbbb1a000)
	libjansson.so.4 => /lib/x86_64-linux-gnu/libjansson.so.4 (0x00007fddbbb0a000)
	libzbar.so.0 => /lib/x86_64-linux-gnu/libzbar.so.0 (0x00007fddbbaca000)
	libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007fddbbaba000)
	libprotobuf-c.so.1 => /lib/x86_64-linux-gnu/libprotobuf-c.so.1 (0x00007fddbbaaa000)
	libsecret-1.so.0 => /lib/x86_64-linux-gnu/libsecret-1.so.0 (0x00007fddbba42000)
	libqrencode.so.4 => /lib/x86_64-linux-gnu/libqrencode.so.4 (0x00007fddbba32000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fddbb84a000)
	libgdk-3.so.0 => /lib/x86_64-linux-gnu/libgdk-3.so.0 (0x00007fddbb73a000)
	libgmodule-2.0.so.0 => /lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007fddbb732000)
	libpangocairo-1.0.so.0 => /lib/x86_64-linux-gnu/libpangocairo-1.0.so.0 (0x00007fddbb71a000)
	libpango-1.0.so.0 => /lib/x86_64-linux-gnu/libpango-1.0.so.0 (0x00007fddbb6aa000)
	libharfbuzz.so.0 => /lib/x86_64-linux-gnu/libharfbuzz.so.0 (0x00007fddbb5a2000)
	libcairo.so.2 => /lib/x86_64-linux-gnu/libcairo.so.2 (0x00007fddbb47a000)
	libpangoft2-1.0.so.0 => /lib/x86_64-linux-gnu/libpangoft2-1.0.so.0 (0x00007fddbb45a000)
	libfontconfig.so.1 => /lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007fddbb40a000)
	libfribidi.so.0 => /lib/x86_64-linux-gnu/libfribidi.so.0 (0x00007fddbb3ea000)
	libcairo-gobject.so.2 => /lib/x86_64-linux-gnu/libcairo-gobject.so.2 (0x00007fddbb3da000)
	libatk-1.0.so.0 => /lib/x86_64-linux-gnu/libatk-1.0.so.0 (0x00007fddbb3aa000)
	libepoxy.so.0 => /lib/x86_64-linux-gnu/libepoxy.so.0 (0x00007fddbb27a000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fddbb19a000)
	libXi.so.6 => /lib/x86_64-linux-gnu/libXi.so.6 (0x00007fddbb182000)
	libX11.so.6 => /lib/x86_64-linux-gnu/libX11.so.6 (0x00007fddbb03a000)
	libatk-bridge-2.0.so.0 => /lib/x86_64-linux-gnu/libatk-bridge-2.0.so.0 (0x00007fddbaffa000)
	libXfixes.so.3 => /lib/x86_64-linux-gnu/libXfixes.so.3 (0x00007fddbaff2000)
	libjpeg.so.62 => /lib/x86_64-linux-gnu/libjpeg.so.62 (0x00007fddbaf5a000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fddbaf3a000)
	libmount.so.1 => /lib/x86_64-linux-gnu/libmount.so.1 (0x00007fddbaed2000)
	libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007fddbaea2000)
	libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8 (0x00007fddbae92000)
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007fddbadf2000)
	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007fddbadca000)
	libdbus-1.so.3 => /lib/x86_64-linux-gnu/libdbus-1.so.3 (0x00007fddbad72000)
	libv4l2.so.0 => /lib/x86_64-linux-gnu/libv4l2.so.0 (0x00007fddbad62000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fddbc8c2000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fddbad5a000)
	libxkbcommon.so.0 => /lib/x86_64-linux-gnu/libxkbcommon.so.0 (0x00007fddbad12000)
	libwayland-client.so.0 => /lib/x86_64-linux-gnu/libwayland-client.so.0 (0x00007fddbacfa000)
	libwayland-cursor.so.0 => /lib/x86_64-linux-gnu/libwayland-cursor.so.0 (0x00007fddbacea000)
	libwayland-egl.so.1 => /lib/x86_64-linux-gnu/libwayland-egl.so.1 (0x00007fddbace2000)
	libXext.so.6 => /lib/x86_64-linux-gnu/libXext.so.6 (0x00007fddbacca000)
	libXcursor.so.1 => /lib/x86_64-linux-gnu/libXcursor.so.1 (0x00007fddbacba000)
	libXdamage.so.1 => /lib/x86_64-linux-gnu/libXdamage.so.1 (0x00007fddbacb2000)
	libXcomposite.so.1 => /lib/x86_64-linux-gnu/libXcomposite.so.1 (0x00007fddbacaa000)
	libXrandr.so.2 => /lib/x86_64-linux-gnu/libXrandr.so.2 (0x00007fddbac9a000)
	libXinerama.so.1 => /lib/x86_64-linux-gnu/libXinerama.so.1 (0x00007fddbac92000)
	libthai.so.0 => /lib/x86_64-linux-gnu/libthai.so.0 (0x00007fddbac82000)
	libfreetype.so.6 => /lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007fddbabb2000)
	libgraphite2.so.3 => /lib/x86_64-linux-gnu/libgraphite2.so.3 (0x00007fddbab82000)
	libpixman-1.so.0 => /lib/x86_64-linux-gnu/libpixman-1.so.0 (0x00007fddbaad2000)
	libxcb-shm.so.0 => /lib/x86_64-linux-gnu/libxcb-shm.so.0 (0x00007fddbaaca000)
	libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007fddbaa9a000)
	libxcb-render.so.0 => /lib/x86_64-linux-gnu/libxcb-render.so.0 (0x00007fddbaa8a000)
	libXrender.so.1 => /lib/x86_64-linux-gnu/libXrender.so.1 (0x00007fddbaa7a000)
	libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fddbaa4a000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fddbaa42000)
	libatspi.so.0 => /lib/x86_64-linux-gnu/libatspi.so.0 (0x00007fddbaa02000)
	libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007fddba9aa000)
	libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007fddba8da000)
	libv4lconvert.so.0 => /lib/x86_64-linux-gnu/libv4lconvert.so.0 (0x00007fddba85a000)
	libdatrie.so.1 => /lib/x86_64-linux-gnu/libdatrie.so.1 (0x00007fddba84a000)
	libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007fddba83a000)
	libXau.so.6 => /lib/x86_64-linux-gnu/libXau.so.6 (0x00007fddba832000)
	libXdmcp.so.6 => /lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007fddba600000)
	libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007fddba822000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fddba5ca000)
	libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007fddba50a000)
	liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007fddba4e2000)
	libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007fddba4ba000)
	libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007fddba4a2000)
	libmd.so.0 => /lib/x86_64-linux-gnu/libmd.so.0 (0x00007fddba812000)

[marcopaganini]

Also, do you have a list of library versions that you're using? I'd like to compare them.

[marcopaganini]

OK, and after two oneliners, I have a list of all libraries and their respective (Debian) versions. Maybe comparing the version numbers (without the debian modifiers) would be illuminating.

libatk1.0-0:          2.46.0-5
libatk-bridge2.0-0:   2.46.0-5
libatspi2.0-0:        2.46.0-5
libbaseencode1:       1.0.15-1
libblkid1:            2.38.1-4
libbrotli1:           1.0.9-2+b6
libbsd0:              0.11.7-2
libcairo2:            1.16.0-7
libcairo-gobject2:    1.16.0-7
libcotp12:            1.2.8-1
libdatrie1:           0.2.13-2+b1
libepoxy0:            1.5.10-1
libffi8:              3.4.4-1
libfontconfig1:       2.14.1-3
libfreetype6:         2.12.1+dfsg-4
libfribidi0:          1.0.8-2.1
libgcrypt20:          1.10.1-3
libgdk-pixbuf-2.0-0:  2.42.10+dfsg-1+b1
libglib2.0-0:         2.74.5-1
libglib2.0-dev:       2.74.5-1
libgraphite2-3:       1.3.14-1
libgtk-3-0:           3.24.36-2
libharfbuzz0b:        6.0.0-1
libjansson4:          2.14-2
libjpeg62-turbo:      1:2.1.2-1+b1
liblz4-1:             1.9.4-1
libmd0:               1.0.4-2
libmount1:            2.38.1-4
libpango-1.0-0:       1.50.12+ds-1
libpangocairo-1.0-0:  1.50.12+ds-1
libpangoft2-1.0-0:    1.50.12+ds-1
libpcre2-8-0:         10.42-1
libpixman-1-0:        0.42.2-1
libpng16-16:          1.6.39-2
libprotobuf-c1:       1.4.1-1+b1
libqrencode4:         4.1.1-1
libsecret-1-0:        0.20.5-3
libsystemd0:          252.5-2
libthai0:             0.1.29-1
libuuid1:             2.38.1-4
libv4l-0:             1.22.1-5+b1
libv4lconvert0:       1.22.1-5+b1
libwayland-client0:   1.21.0-1
libwayland-cursor0:   1.21.0-1
libwayland-egl1:      1.21.0-1
libx11-6:             2:1.8.3-3
libxau6:              1:1.0.9-1
libxcb1:              1.15-1
libxcb-render0:       1.15-1
libxcb-shm0:          1.15-1
libxcomposite1:       1:0.4.5-1
libxcursor1:          1:1.2.1-1
libxdamage1:          1:1.1.6-1
libxdmcp6:            1:1.1.2-3
libxext6:             2:1.3.4-1+b1
libxfixes3:           1:6.0.0-2
libxi6:               2:1.8-1+b1
libxinerama1:         2:1.1.4-3
libxkbcommon0:        1.5.0-1
libxrandr2:           2:1.5.2-2+b1
libxrender1:          1:0.9.10-1.1
libzbar0:             0.23.92-7
libzstd1:             1.5.2+dfsg2-3

[paolostivanin]

The most important stuff (according to the type of bug you're facing) is:

• libjansson
• libgcrypt
• gcc
• glib2

And we both are using the same versions (2.14, 1.10.1, 12.2, 2.74.5). I am on openSUSE Tumbleweed.
I looked at patches that Debian could have applied, but I don't see anything special there.

I just installed Debian testing and I can reproduce the bug... 😲

[paolostivanin]

I can see that, for whatever reason, there's an extract char at the of the input:

        }
    ]
}P\ufffd?

[paolostivanin]

Bug should be fixed now. If you try latest master, you should be able to import your data.
Now I'm gonna test it on other distros and, if everything looks good, I'll release a new version today or tomorrow.