Fortigate Backups Timing Out

[French-Josh]

Fortigate Backups Timing Out

[Luanpablo100]

Hi Josh!

Have you tried applying the oxidized debug configuration?
debug: true

This way we can better analyze what is happening when oxidized tries to connect to your Fortigate.

[French-Josh]

Thanks for the quick response!

I have done so, I can see that the SSH appears to get in and sends through the commands to get the configs out. But after that there are just loads of "2 jobs running in parallel", obviously thats the two fortigates that haven't yet succeeded.

Nov 23 13:24:53 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:24:53.396631 #2017283] DEBUG -- : resolving DNS for FORTIGATE...
Nov 23 13:24:57 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:24:57.251800 #2017283] DEBUG -- : lib/oxidized/model/model.rb Executing show inventory | no-more
Nov 23 13:24:59 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:24:59.765683 #2017283] DEBUG -- : lib/oxidized/worker.rb: Added /FORTIGATE to the job queue
Nov 23 13:24:59 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:24:59.766136 #2017283] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for FORTIGATE at 2023-11-23 13:24:59 UTC
Nov 23 13:24:59 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:24:59.767265 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: Connecting to FORTIGATE
Nov 23 13:24:59 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:24:59.912108 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.]+(\s[(\w-.)]+)??\s?[#>$]\s?)$/] at FORTIGATE
Nov 23 13:25:00 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:00.730706 #2017283] DEBUG -- : lib/oxidized/input/cli.rb: Running post_login commands at FORTIGATE
Nov 23 13:25:00 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:00.730926 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb get system status @ FORTIGATE with expect: /^([-\w.]+(\s[(\w-.)]+)??\s?[#>$]\s?)$/
Nov 23 13:25:00 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:00.731222 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.]+(\s[(\w-.)]+)??\s?[#>$]\s?)$/] at FORTIGATE
Nov 23 13:25:01 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:01.148704 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb config global @ FORTIGATE with expect: /^([-\w.]+(\s[(\w-.)]+)??\s?[#>$]\s?)$/
Nov 23 13:25:01 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:01.148895 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.]+(\s[(\w-.)]+)??\s?[#>$]\s?)$/] at FORTIGATE
Nov 23 13:25:01 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:01.551642 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb get system ha status @ FORTIGATE with expect: /^([-\w.]+(\s[(\w-.)]+)??\s?[#>$]\s?)$/
Nov 23 13:25:01 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:01.551898 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.]+(\s[(\w-.)]+)??\s?[#>$]\s?)$/] at FORTIGATE
Nov 23 13:25:01 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:01.957126 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb get hardware status @ FORTIGATE with expect: /^([-\w.]+(\s[(\w-.)]+)??\s?[#>$]\s?)$/
Nov 23 13:25:01 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:01.957359 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.]+(\s[(\w-.)]+)??\s?[#>$]\s?)$/] at FORTIGATE
Nov 23 13:25:02 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:02.362177 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb end @ FORTIGATE with expect: /^([-\w.]+(\s[(\w-.)]+)??\s?[#>$]\s?)$/
Nov 23 13:25:02 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:02.362567 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.]+(\s[(\w-.)]+)??\s?[#>$]\s?)$/] at FORTIGATE
Nov 23 13:25:02 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:02.796538 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb show full-configuration | grep . @ FORTIGATE with expect: /^([-\w.]+(\s[(\w-.)]+)??\s?[#>$]\s?)$/
Nov 23 13:25:02 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:02.796691 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.]+(\s[(\w-.)]+)??\s?[#>$]\s?)$/] at FORTIGATE

Is there perhaps a better spot in the config to put the debug that would give me a better idea of what is failing, because this seems to suggest the login is working and its simply not reading the returned data.

I have also tested the regex and it seems to be fine as well.

Nov 23 13:35:02 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:35:02.972196 #2017283] DEBUG -- : lib/oxidized/node.rb: Oxidized::SSH failed for FORTIGATE

The above would seem to suggest that its SSH related. But I have tested SSH using the creds in the configs as well as the fortigates that are working have the same creds configured. Is it timing out perhaps because it cannot get the expected data back in time?

EDIT: in the .config/oxidized/logs/ dir I can see the IP of the fortigates with configs in them. Is this perhaps something else?

Thanks in advance.

[Luanpablo100]

If you open the file located at .config/oxidized/logs/{Fortigate-IP}-ssh you can see where exactly oxidized is "hanging". In my opinion, it presents a better visualization than the oxidized service logs.
If possible, could you share the result here? Of course, not displaying your Fortigate's settings, only the commands executed, and error messages, if any.

[French-Josh]

So I have actually had this in my first deployment as well, in that case I removed the file, removed the device in Libre and readded everything and thats when it started to work (in my testing I mean). So I removed the files this time again to try that again, unfortunately no such luck this time.

I did get an output of the files before I removed them though. It appears that its getting to the last command and exiting.

Last entry in the log file is:

FORTIGATE $ exit

Currently trying to get it to regenerate the files, but yet to have any luck. Do you think its worth making the timeout something silly high like 20-30 mins?

[French-Josh]

So I suspect I have semi figured out the issue but some clarification for it would be nice.

Essentially what seems to be the issue is that each of the larger config dumps takes roughly 6 minutes to be processed.

I have been testing with two Fortigates configured and when I set it down to just the one it work and the configs are taken (as I mentioned the completion for the Fortigate is normally 6 minutes into the run).

I increased the overall timeout for run to 960 seconds and added the second Fortigate back. As expected at roughly 12 minutes, both Fortigates configs complete successfully. I tried changing the threads to 60 from 30 assuming that perhaps it was taking to long to run each scan. Is there another setting I should be looking to change to get this down? I have more Fortigates that have about the same amount of lines in config, so making the timeout close to 30 minutes seems a little high. Unless this is expected behaviour?

Any info to clarify would be great!

[robertcheramy]

Possible duplicate of #1699, fixed should come with PR #2935.