New onlinebanking renamed CSV fields

[siddhantgoel]

New onlinebanking renamed CSV fields

[siddhantgoel]

@nils-werner I just published v0.16.0 on PyPI which should be able to work with the new format. Let me know if something doesn't work. It looks like DKB hasn't added date filtering to their new CSV exports so I will be sticking to the older interface for now for personal use.

[siddhantgoel]

Ugh, they seem to keep changing things. At the moment I'm unfortunately not able to work much due to health reasons. I would suggest in the meanwhile to continue using the older online banking. I'll try to get to this once I'm able to work normally again. Alternatively I'm happy to review and merge PRs.

[tbm]

I get:

TypeError: ECImporter.__init__() got an unexpected keyword argument 'file_encoding'

Maybe the file_encoding parameter could still be accepted and ignored (or lead to a deprecated warning)? In any case, the removal should be mentioned in CHANGELOG.

After removing that, it works as expected for Cash and Tagesgeld. (I don't have the credit card anymore). thank you!

[jjungnickel]

While I can confirm that both Cash and Tagesgeld work, I, too no longer have a DKB credit card

[siddhantgoel]

I've been expecting this to come up since they announced the redesign. 😅

Thankfully, they didn't switch off the older banking just yet so I presume it's not time critical. Regardless, I'll try to push a fix for this hopefully this weekend (if not then most likely the coming week).

[siddhantgoel]

Btw, I couldn't find a way in the new online banking to export CSV transactions for a specific timeframe. At the moment you get a CSV file with all the currently displayed transactions.

I wonder if I'm unable to find it or they don't support it yet?

[nils-werner]

Hmm, It doesn't seem to work for me. The output of

bean-extract -e personal.beancount personal.import girokonto.csv

is empty and

bean-file personal.import girokonto.csv

doesn't do anything either...

Do I have to update my importer config?

[siddhantgoel]

You shouldn't have to update any configs. And it should generally work (I tested it locally with a few downloads), unless they changed their format again. 🤷🏼‍♂️

Are you able to send me a few lines from the CSV file you're working with after removing any personal details?

[nils-werner]

Of course!

"Kontonummer:";"DE11111111111111111111 / Girokonto";

"Von:";"03.08.2023";
"Bis:";"02.10.2023";
"Kontostand vom 02.10.2023:";"1.111,11 EUR";

"Buchungstag";"Wertstellung";"Buchungstext";"Auftraggeber / Beg�nstigter";"Verwendungszweck";"Kontonummer";"BLZ";"Betrag (EUR)";"Gl�ubiger-ID";"Mandatsreferenz";"Kundenreferenz";
"02.10.2023";"02.10.2023";"Kartenzahlung";"ALDI SUED";"2023-09-30      Debitk.11 VISA Debit";"11111111111111111111";"BYLADEM1001";"-16,45";"";"";"111111111111111";

and

"Kreditkarte:";"1111********1111";

"Zeitraum:";"letzten 60 Tage";
"Saldo:";"1111.11 EUR";
"Datum:";"29.09.2023";

"Umsatz abgerechnet und nicht im Saldo enthalten";"Wertstellung";"Belegdatum";"Beschreibung";"Betrag (EUR)";"Urspr�nglicher Betrag";
"Ja";"23.09.2023";"22.09.2023";"HabenzinsenZ 000001111 T 030   0000";"1,11";"";
"Ja";"23.08.2023";"22.08.2023";"HabenzinsenZ 000001111 T 031   0000";"1,11";"";

[siddhantgoel]

Hmm, those entries to be from the older online banking. Ignoring the umlauts (I assume that was a copy paste thing when pasting into Github comments) I wrote the following test cases for these:

def test_ec(tmp_file):
    tmp_file.write_text(
        _format(
            """
            "Kontonummer:";"{iban} / Girokonto";

            "Von:";"03.08.2023";
            "Bis:";"02.10.2023";
            "Kontostand vom 02.10.2023:";"1.111,11 EUR";

            {header}
            "02.10.2023";"02.10.2023";"Kartenzahlung";"ALDI SUED";"2023-09-30      Debitk.11 VISA Debit";"11111111111111111111";"BYLADEM1001";"-16,45";"";"";"111111111111111";
            """,
            dict(iban=IBAN, header=HEADER),
        )
    )

    importer = ECImporter(IBAN, "Assets:DKB:EC", file_encoding="utf-8")

    with tmp_file.open() as fd:
        directives = importer.extract(fd)

    assert len(directives) == 2

def test_credit(tmp_file):
    prefix = CARD_NUMBER[:4]
    suffix = CARD_NUMBER[-4:]

    tmp_file.write_text(
        _format(
            """
            "Kreditkarte:";"{prefix}********{suffix}";

            "Zeitraum:";"letzten 60 Tage";
            "Saldo:";"1111.11 EUR";
            "Datum:";"29.09.2023";

            {header}
            "Ja";"23.09.2023";"22.09.2023";"HabenzinsenZ 000001111 T 030   0000";"1,11";"";
            "Ja";"23.08.2023";"22.08.2023";"HabenzinsenZ 000001111 T 031   0000";"1,11";"";
            """,
            dict(prefix=prefix, suffix=suffix, header=HEADER),
        )
    )

    importer = CreditImporter(CARD_NUMBER, "Assets:DKB:Credit")

    with tmp_file.open() as fd:
        directives = importer.extract(fd)

    assert len(directives) == 3

... and they seem to pass. 😕

Did you mean the CSV entries from the newer online banking don't work or from the older one?

[nils-werner]

OK I'm confused now where I got those CSV's from... the following two are definitely from the new banking and they don't seem to work:

"Konto";"Girokonto DE0XXXXXXXXXXXXXXXXXX"
""
"Kontostand vom 03.11.2023:";"XXXX,XX EUR"
""
"Buchungsdatum";"Wertstellung";"Status";"Zahlungspflichtige*r";"Zahlungsempfänger*in";"Verwendungszweck";"Umsatztyp";"Betrag";"Gläubiger-ID";"Mandatsreferenz";"Kundenreferenz"
"03.11.23";"";"Vorgemerkt";"ISSUER";"ALDI";"2023-11-03T09:48 Debitk. 0 2099-12 Zahl.System VISA De bit     (POS)";"Ausgang";"-3,18 €";"";"";"XXXXXXXXXXXXXX"
"03.11.23";"03.11.23";"Gebucht";"ISSUER";"REWE";"2023-11-02 Debitk.XX VISA Debit";"Ausgang";"-9,32 €";"";"";"XXXXXXXXXXXXXXXXX"
"03.11.23";"03.11.23";"Gebucht";"ISSUER";"XXXXX";"2023-11-02 Debitk.XX VISA Debit";"Ausgang";"-50,00 €";"";"";"XXXXXXXX"

"Karte";"Notgroschen XXXXXXXXXXXXX"
""
"Saldo vom 03.11.2023:";"--XXXX,X EUR"
""
"Belegdatum";"Wertstellung";"Status";"Beschreibung";"Umsatztyp";"Betrag";"Fremdwährungsbetrag"
"20.10.23";"21.10.23";"Gebucht";"Habenzinsen";"Umsatztyp unbekannt";"X,XX €";""
"22.09.23";"23.09.23";"Gebucht";"Habenzinsen";"Umsatztyp unbekannt";"X,XX €";""

one issue I see is that Girokonto and Notgroschen are names that can be individualized by the user, so they shouldn't be hardcoded in self._expected_header_regex

[jjungnickel]

Even worse, in more recent CSVs, the identifying headers are missing completely and there's a single empty line instead:

""
"Kontostand vom 15.12.2023:";"XXXX,XX EUR"
""
"Buchungsdatum";"Wertstellung";"Status";"Zahlungspflichtige*r";"Zahlungsempfänger*in";"Verwendungszweck";"Umsatztyp";"IBAN";"Betrag (€)";"Gläubiger-ID";"Mandatsreferenz";"Kundenreferenz"

[siddhantgoel]

I just published v0.18.0b1 (marked as pre-release for now) which should be able to handle files from the old and new banking interfaces.

I had to get rid of the file_encoding parameter because the older files were always ISO-8859-1 encoded and the newer ones are UTF-8 encoded (with byte order mark at the beginning) which makes it easier to decide how to load the files. You can find more details about the changes in #147.

It would be super helpful if you could try this version out and let me know if it works for you. Thanks!

/cc @tbm

[tbm]

[removed comment about Tagesgeld not working]

[tbm]

I added a (now removed) comment about Tagesgeld not working. Please ignore that. My CSV was corrupt (probably a combination of the old and new CSV file). I've downloaded a new CSV from DKB and it works as expected.

So both EC and Tagesgeld work. I can't test the credit card.

[siddhantgoel]

Awesome, thanks so much for testing!

@nils-werner @jjungnickel would you be able to test out the credit card exports? If so, that would be quite helpful.

[siddhantgoel]

Thanks for testing! I've just published 0.18.0 as a normal release and will close this issue. If something else comes up feel free to reopen.