Data Importer stops Autoimport of multiple Nordigen/GoCardless files if one has zero transactions

[rerobade]

Data Importer stops Autoimport of multiple Nordigen/GoCardless files if one has zero transactions

[JC5]

I'm not sure why you see it as an error to get no transactions back

That's not entirely what's happening in the code, but I understand the confusion. I'll pick it up!

[JC5]

Thanks all. This will be fixed in the next release. You can test if this works by running :develop.

[JC5]

Nice, i see it too. Check out the latest :develop build, it should self-report as build #501:

Script:        1.6 (2022-02-17)
User:          'root'
Group:         'root'
Working dir:   '/var/www/html'
Build number:  501
Build date:    17-01-2024 18:09:21 UTC

[JC5]

Lets close this when 1.4.2 is released. The other issue is on my list but not very high.

[rerobade]

I'm not sure why you see it as an error to get no transactions back

That's not entirely what's happening in the code, but I understand the confusion. I'll pick it up!

Thank you :-D

[thomase1234]

Hi @JC5 ,
I have an issue which is triggered under the same conditions.
My first json import file will fetch multiple transactions. The 2nd json fetches zero. Still the "Import Report Mail" of the 2nd will show all transactions ( obviously as duplicates ) of the 1st.

First report:

2nd report:

The fact that transaction 5155 is twice listed as duplicate is normal. But it's not possible that the bank in the 2nd json has an identical list of transaction as the first json.

I have a 3rd and 4th json but those are never processed, because the process stopped with the same error as the original poster. "Too many errors in the data conversion"

Could it be that the transactions list is not cleared when switching from 1 json configuration file to another ?

If this issue is completely unrelated, I can open a new issue.

Thanks,
Thomas

[rerobade]

Hi @JC5 ,

thanks for the quick fix, the import works again with the latest :develop version. But the defect reported by @thomase1234 above still persists. The first Mail with an error like

Line #1: Could not download from GoCardless: GuzzleHttp\Exception\ServerException: Server error: GET https://ob.nordigen.com/api/v2/accounts/4aa8ddb5-aaa8-4f9e-bdeb-abc845bf729e/transactions/ resulted in a 503 Service Unavailable response: {"summary":"Couldn't update account transactions","detail":"Institution service unavailable","status_code":503,"type":"S (truncated...)

also contains the successful messages from the previous import. So it looks like some list or array is not reseted after an import. It's up to you of this is a separate issue or something you wanted also to fix with this defect.
And I'm also not so sure if an exception in a Mail will make some users nervous and causes unwanted bug-reports in your inbox :-)

Thanks and regards
René

[JC5]

Thanks René. Tell me, how did you fire the import routine that resulted in this error? Was it a cron job, the UI, something else? Please share the command as detailed as you are able, it helps me narrow down the search.

[rerobade]

Hi @JC5,

I've a simple daily cron-job which just calls
curl --location --request POST 'http://fidi:8080/autoimport?directory=/configurations&secret=<my secret>'
And this is the configuration of my importer container within my docker-compose.yml

...
fidi:
    container_name: fireflyiii_fidi
    image: fireflyiii/data-importer:develop
    restart: unless-stopped
    networks:
      - fireflyiii
    environment:
      - FIREFLY_III_URL=http://fireflyiii_app:8080
      - VANITY_URL=https://${DNS_NAME}
      - FIREFLY_III_ACCESS_TOKEN_FILE=/run/secrets/fireflyiii_access_token
      - NORDIGEN_ID_FILE=/run/secrets/fireflyiii_nordigen_id
      - NORDIGEN_KEY_FILE=/run/secrets/fireflyiii_nordigen_key
      - TZ=${TZ}
      - JSON_CONFIGURATION_DIR=/configurations
      - AUTO_IMPORT_SECRET_FILE=/run/secrets/fireflyiii_auto_import_secret
      - CAN_POST_AUTOIMPORT=true
      - CAN_POST_FILES=true
      - IMPORT_DIR_ALLOWLIST=/configurations
      - ENABLE_MAIL_REPORT=true
      - [email protected]
      - MAIL_MAILER=smtp
      - MAIL_HOST_FILE=/run/secrets/smtp_host
      - MAIL_PORT_FILE=/run/secrets/smtp_port
      - MAIL_FROM_ADDRESS_FILE=/run/secrets/smtp_mail_from
      - MAIL_USERNAME_FILE=/run/secrets/smtp_name
      - MAIL_PASSWORD_FILE=/run/secrets/smtp_password
      - MAIL_ENCRYPTION_FILE=/run/secrets/smtp_secure
      - STATIC_CRON_TOKEN_FILE=/run/secrets/fireflyiii_static_cron_token
    volumes:
      - ./fidi-config:/configurations:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - xxxxxx:8080
    depends_on:
      - app
    secrets:
      - fireflyiii_access_token
      - fireflyiii_nordigen_id
      - fireflyiii_nordigen_key
      - fireflyiii_auto_import_secret
      - fireflyiii_static_cron_token
      - smtp_host
      - smtp_port
      - smtp_mail_from
      - smtp_name
      - smtp_password
      - smtp_secure
...    

Regards René

[rerobade]

Hi @JC5,

the duplicated messages are also solved now, so I will close the defect from my perspective. Should I open a Feature Request for the not so nice looking mail with exceptions or do you have it already on your agenda?

Regards René

[github-actions]

Hi there! This is an automatic reply. Share and enjoy

This issue is now 🔒 closed. Please be aware that closed issues are not monitored by the developer of Firefly III.

• If the original bug is not actually fixed, please open a new issue. Refer to this issue for clarity.
• Follow-up questions must be posted in a new discussion
• Further replies to this issue may get no response.

If there is more to discuss, please open a new issue or discussion.

Thank you for your contributions.