spotify/pyfg

pyfg's Issues

pyFG does not handle "move" command under "config firewall policy" correctly?

Hello guys,

I recently tried to use pyFG to config Fortigate (5.4.x firmware). It is good for the "config" and "edit" cmd.
However, when I trying to use "move" cmd under "config firewall policy" to re-order a policy, It is found that the compare_config() method cannot return the changed config.
As a result the diff cannot be committed.

Below are the cmds that I would like to execute, to move the policy id 611 after id 999.
'move_policy.txt'

config firewall policy
move 611 after 999
end

This is the script that I modify from "https://github.com/spotify/pyfg/blob/master/examples/example5.py" to execute the above move_policy.txt to the 'vpn' vdom

#!/usr/bin/env python

# Gets router bgp config from the device, then do some changes to the BGP parameters, deletes a neighbor,
# creates a new one, modifies another and computes the difference
from pyFG import FortiOS
import logging

host = '1.1.1.1'
vdom = 'vpn'
cmdfile = 'move_policy.txt'
user = 'admin'
passwd = 'password'

if __name__ == '__main__':
    f = open(cmdfile, 'r')
    candidate = f.read()
    f.close()

    print "*** This is the candidate configs:"
    print candidate
    print "\n"

    d = FortiOS(hostname=host, vdom=vdom, username=user, password=passwd)
    d.open()
    d.load_config(config_text=candidate, in_candidate=True)

    print "*** This is the diff of the conigs: (compare_config(text=True))"
    print (d.compare_config(text=True))
    print "\n"

    print "*** This is how to reach the desired state: (compare_config())"
    config_changes = d.compare_config()
    print config_changes

    print "*** Result of applying the changes: (d.commit)"
    print d.commit(config_changes, force=True)

    d.close()

This is the output of the result,
the "move" cmd cannot be compared such that nothing can be committed.
The move 611 after 999 cmd cannot found in diff

*** This is the candidate configs:
config firewall policy
move 611 after 999
end


*** This is the diff of the configs: (compare_config(text=True))
+     config firewall policy
+     end


*** This is how to reach the desired state: (compare_config())

*** Result of applying the changes: (d.commit)
None

Process finished with exit code 0

Are are any suggestion to execute "move" command in firewall policy using PyFG?
thank you.

pip install pyfg failing

I am trying to install pyfg using pipenv (python 3.9) but the command fails with the following error:

>pipenv install pyfg

Installing pyfg…
Error:  An error occurred while installing pyfg!
Error text: Collecting pyfg
  Using cached pyfg-0.50.tar.gz (9.1 kB)

    ERROR: Command errored out with exit status 1:
     command: /Users/vikrant.tyagi/.virtualenvs/fortios-automation-uy-maohS/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/kh/m16w2sgn10jd06jfxd58wl_c_2h4vh/T/pip-install-_8rwn0my/pyfg/setup.py'"'"'; __file__='"'"'/private/var/folders/kh/m16w2sgn10jd06jfxd58wl_c_2h4vh/T/pip-install-_8rwn0my/pyfg/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/kh/m16w2sgn10jd06jfxd58wl_c_2h4vh/T/pip-pip-egg-info-c6kf6k4d
         cwd: /private/var/folders/kh/m16w2sgn10jd06jfxd58wl_c_2h4vh/T/pip-install-_8rwn0my/pyfg/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/kh/m16w2sgn10jd06jfxd58wl_c_2h4vh/T/pip-install-_8rwn0my/pyfg/setup.py", line 4, in <module>
        from pip.req import parse_requirements
    ModuleNotFoundError: No module named 'pip.req'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

This is likely caused by a bug in pyfg. Report this to its maintainers.
✘ Installation Failed

Character '#' in configuration is not being handled properly

If the configuration contains the character '#' the script will fail with the following traceback:

Traceback (most recent call last):
  File "./pyFG-test.py", line 16, in <module>
    d.load_config(sys.argv[3])
  File "build/bdist.linux-x86_64/egg/pyFG/fortios.py", line 215, in load_config
  File "build/bdist.linux-x86_64/egg/pyFG/forticonfig.py", line 377, in parse_config_output
AttributeError: 'NoneType' object has no attribute 'get_block_names'

It seems that the character '#' is considered harmful for most parts of the configuration, usually trying to use it fails in the following way:

 # config firewall address
 (address) # edit "test#"

The string contains XSS vulnerability characters

value parse error before 'test#'
Command fail. Return code -173

At least vpn ssl web user-bookmark allows the character:

 # config vpn ssl web user-bookmark
 (user-bookmark) # edit "test#"
new entry 'test#' added
 (test#) # end
 show vpn ssl web user-bookmark
config vpn ssl web user-bookmark
    edit "test#"
    next
end

I actually see no reason why that character is part of the configuration and I will most likely remove it. There also is a chance of this being a FortiOS issue in case this character should not be allowed at all in any part of the configuration. Maybe you can and want to fix this, otherwise it is at least documented.

Files validation

Hi there!

First of all, thank you so much for this awesome API you guys nailed.

I've got a question. Is there any way this API could be used for configuration files validation? I have this project in which I need to make sure no invalid FortiGate configuration file is uploaded (that is, sections without an 'end' or 'next', weird characters...).

Bug with python2.6: AttributeError: 'tuple' object has no attribute 'major'

Lib is not working with python 2.6 & centos 6:

from pyFG import py23_compat
File "/usr/lib/python2.6/site-packages/pyFG/py23_compat.py", line 7
PY2 = sys.version_info.major == 2
AttributeError: 'tuple' object has no attribute 'major'

This is a common bug, PR is comming

Non-standard SSH Port

It looks like FortiOS.open() has no option to specify a port parameter yet paramiko.SSHClient has port parameter enabled.

d.load_config('user adgrp') fail in one vdom

Hi

We have problem load full-config in one vdom.

The fortigate/vdom work as expected and we have found d.load_config('user adgrp') fails as well for all groups. As we have 4563 adgrp it is hard to say were it fail. So we think there is one ore more group that make it fail. a quick check do not find any group without "set"

get system status

Version: FortiGate-1000C v5.2.4,build0688,150722 (GA)

pip show pyfg

Name: pyfg Version: 0.47

[chris@automata pyfg]$ python fg_backup.py fnet01 vdom01
Traceback (most recent call last):
File "fg_backup.py", line 16, in
d.load_config('user adgrp')
File "/usr/local/lib/python2.7/site-packages/pyFG/fortios.py", line 220, in load_config
self.running_config.parse_config_output(config_text)
File "/usr/local/lib/python2.7/site-packages/pyFG/forticonfig.py", line 383, in parse_config_output
current_block.set_param(parameter, ' '.join(data))
AttributeError: 'NoneType' object has no attribute 'set_param'
[chris@automata01 pyfg]$ more fg_backup.py

!/usr/bin/env python

Gets the entire Fortigate config and prints it

from pyFG import FortiOS
import sys

if name == 'main':
hostname = sys.argv[1]
vdom = sys.argv[2]

d = FortiOS(hostname, vdom=vdom, username='admin', password='password')
d.open()
d.load_config('user adgrp')
#d.load_config('full-configuration')
print d.running_config.to_text()
d.close()

[chris@automata pyfg]$

Status of the project?

Hello,

it seems that this project is abandoned. Maybe you could give another person write-access so it is possible to merge the PRs and upload the latest version to PyPI, please.

Paramiko disconnects with EOFError

Hi

I tried to connect to a FG with plan basic Paramiko and struggled. I then tested with your module and basically got the same error:

from pyFG import FortiOS
d = FortiOS('10.8.20.102')
d.open()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/pyFG/fortios.py", line 94, in open
    self.ssh.connect(**cfg)
  File "build/bdist.macosx-10.11-intel/egg/paramiko/client.py", line 325, in connect
    t.start_client()
  File "build/bdist.macosx-10.11-intel/egg/paramiko/transport.py", line 492, in start_client
    raise e
EOFError

Any idea what this could be? It looks like it's only happening with FortiOS Devices, I can connect to normal Linux hosts without any issues.

ImportError: No module named req

Install error:

Collecting pyFG

  Downloading pyfg-0.50.tar.gz

    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-GOjqC9/pyFG/setup.py", line 4, in <module>
        from pip.req import parse_requirements
    ImportError: No module named req

From upstream, you should not be depending on this pip internal: pypa/pip#3240

pyFG does not handle multi-line values correctly

I recently tried to use pyFG's FortiConfig on a stock 100D (5.4.x firmware) Fortigate config, and found that it would truncate multi-line configurations, such as the certificate lines (which meant the resulting configs would not apply back cleanly).

From the original config:

config vpn certificate local
    edit "Fortinet_CA_SSLProxy"
        set certificate "-----BEGIN CERTIFICATE-----
MIID3jCCAsagAwIBAgIILmc2LyCLCs4wDQYJKoZIhvcNAQEFBQAwgaUxCzAJBgNV
...
OaKXO5cGgRFs5sSQMWunVOlATHhkMe+UQeZstT78oP4s6A==
-----END CERTIFICATE-----"
   next
end

Which after adding to a FortiConfig and calling to_text() produced:

config vpn certificate local
    edit Fortinet_CA_SSLProxy
        set certificate "-----BEGIN CERTIFICATE-----
    next
end

I'm happy to provide further details, but I think the issue is clear enough. The problem is that the parser seems to be line based, and so would need to build up a line memory if it ever doesn't encounter a closing quote (which will add complexity). At the moment it appears that quotes are removed completely, which may cause separate problems with named sections that contain spaces. So there's a fair bit of complexity surrounding handling this correctly. I'm happy to test any potential fixes for this if it would help though?

Slow reading config

I found a problem while reading fortios config. With large config, the read takes more than 15 minutes to complete.

Timeout on discard_config command executing

Question about an issue

Hello there, I am facing an issue of timeout using pyfg in fortios driver for napalm integrated in salt. But the problem does seem to come from here.
When trying to discard_config on a fortios device, the channel raise a socket timeout and the operation fails.

I achieved to make this working perfectly just by changing the timeout value
<...>/python2.7/dist-package/pyFG/fortios.py, line 386:

--chan.settimeout(5)
++chan.settimeout(15)

I dont have tried other timeout values but '5 seconds' failed 30 tries over 30 whereas '15 seconds' worked each 5 times I tried.
I seem to be the only having ever faced this issue, so I suppose it could be a problem on my side.
But I don't see for now what it is, so, question is :
do you see from where on my side the problem could come
would it be interesting to do a pr to change the value, or make it a param.
or do i just go away and change it locally

Thanks

traceback

2018-06-12 15:20:03,453 [/usr/lib/python2.7/dist-packages/salt/utils/napalm.pyc     :229 ][ERROR   ][12137] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/napalm.py", line 167, in call
    out = getattr(napalm_device.get('DRIVER'), method)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/napalm_fortios/fortios.py", line 142, in discard_config
    self.device.load_config(in_candidate=True)
  File "/usr/local/lib/python2.7/dist-packages/pyFG/fortios.py", line 212, in load_config
    config_text = self.execute_command(command)
  File "/usr/local/lib/python2.7/dist-packages/pyFG/fortios.py", line 151, in execute_command
    for e in error_chan.read():
  File "/usr/local/lib/python2.7/dist-packages/paramiko/file.py", line 193, in read
    new_data = self._read(self._DEFAULT_BUFSIZE)
  File "/usr/local/lib/python2.7/dist-packages/paramiko/channel.py", line 1344, in _read
    def _read(self, size):
  File "/usr/local/lib/python2.7/dist-packages/paramiko/channel.py", line 733, in recv_stderr
    raise socket.timeout()
timeout

version

pyfg: 0.50
paramiko: 2.4.1

Version: FortiGate-60C-POE v5.2.13,build0762,171212 (GA)
Virus-DB: 16.00560(2012-10-19 08:31)
Extended DB: 1.00000(2012-10-17 15:46)
IPS-DB: 5.00555(2014-10-07 01:21)
IPS-ETDB: 0.00000(2001-01-01 00:00)
Serial-Number: FG60CP3W13000001
Botnet DB: 1.00000(2012-05-28 22:51)
BIOS version: 04000003
System Part-Number: P12195-01
Log hard disk: Not available
Internal Switch mode: switch
Hostname: fg60cp3w13000001
Operation Mode: NAT
Current virtual domain: root
Max number of virtual domains: 10
Virtual domains status: 1 in NAT mode, 0 in TP mode
Virtual domain configuration: disable
FIPS-CC mode: disable
Current HA mode: standalone
Branch point: 762
Release Version Information: GA
System time: Tue Jun 12 08:38:03 2018

Add FortiConfig.to_dict() for genie parser to compare.

#5adb234

` def to_dict(self, root=None, tree_dict=None):
"""
This method convert FortiConfig to dict.

    Args:
        - **root** (class:`~pyFG.forticonfig.FortiConfig` object):
                * If ``None`` use self as tree root.
        - **tree_dict** (dict): This value is for output.
                * If ``None`` use a blank dict.
    """
    if root is None:
        root = self
    if tree_dict is None:
        tree_dict = {}
    if not hasattr(root, 'get_block_names') or not root.get_block_names():
        return
    for child in root.get_block_names():
        if not tree_dict.get(child):
            if root[child].get_parameter_names():
                tree_dict[child] = {k: root[child].get_param(k).replace('"', '')
                                    for k in root[child].get_parameter_names()}
            else:
                tree_dict[child] = {}
        self.to_dict(root[child], tree_dict[child])
    return tree_dict

`

Add option to ignoring ssl errors

This would be useful for when configuring devices that still only have self-signed certs. A crude workaround was documented on the fortinet developers network page.
https://fndn.fortinet.net/index.php?/forums/topic/1896-examples-of-ansible-28-and-new-fortinet-modules-fortiosapi-in-action/&do=findComment&comment=7294

Commit not working with VDOM

Using the latest version of the pyFG, I'm trying to merge the running config on a fortigate of mine with a new snippet of code. I'm running with VDOMs enabled.

In [1]: import pyFG
In [2]: device = pyFG.FortiOS("fw1", vdom="FW", username="admin", password="password")
In [3]: device.open()
In [4]: device.candidate_config = pyFG.FortiConfig("candidate")
In [5]: device.running_config = pyFG.FortiConfig("running")
In [6]: configuration = open("/Users/tim/code/napalm/generated/fw1.txt").read()
In [7]: device.load_config(in_candidate=True, config_text=configuration)
In [8]: for block in device.candidate_config.get_block_names():
   ...:  device.load_config(path=block, empty_candidate=True)
   ...:
In [12]: print device.compare_config()
    config firewall address
        edit Marist
          set subnet 148.100.0.0 255.255.0.0
        next
    end

The load config looks mostly correct, but is missing the commands to move this into the correct VDOM. So when I try to commit if get an error

In [13]: device.commit()
---------------------------------------------------------------------------
FailedCommit                              Traceback (most recent call last)
<ipython-input-13-764a21633a04> in <module>()
----> 1 device.commit()

/Library/Python/2.7/site-packages/pyFG/fortios.pyc in commit(self, config_text, force)
    277             * :class:`~pyFG.exceptions.ForcedCommit` -- Something failed but we avoided any rollback
    278         """
--> 279         self._commit(config_text, force)
    280
    281     def _commit(self, config_text=None, force=False, reload_original_config=True):

/Library/Python/2.7/site-packages/pyFG/fortios.pyc in _commit(self, config_text, force, reload_original_config)
    332
    333             if exit_code < 0 :
--> 334                 raise exceptions.FailedCommit(wrong_commands)
    335
    336     def rollback(self):

FailedCommit: [('-1', 'edit Marist'), ('-1', 'set subnet 148.100.0.0 255.255.0.0'), ('-1', 'next'), ('-1', 'end')]

Now what is interesting is that if I run compare_config again, I get the correct config snippet it, VDOM and all. I'm also able to commit this just fine.

In [14]: print device.compare_config()
conf vdom
  edit FW
    config firewall address
        edit Marist
          set subnet 148.100.0.0 255.255.0.0
        next
    end
end
In [15]: device.commit()
In [16]:

I noticed this issue when trying to use napalm and this thread started there
napalm-automation-community/napalm-fortios#13

"Error reading running config" while backup config

Hi
I had some some error while backuping configuration on Fortigate 3000.

I have the message "Error reading running config" and a raise socket.timeout() exception.

My investigating found that it was the error channel of the ssh connection which raise this timeout and stop the reading for the standard output.

I do this modification on the plugin python module:
pyfg/pyFG/fortios.py
from line 149 replace this
error = '' output = '' for e in error_chan.read(): error = error + self._read_wrapper(e) for o in output_chan.read(): output = output + self._read_wrapper(o)

by

error = '' output = '' try: for e in error_chan.read(): error = error + self._read_wrapper(e) except: pass for o in output_chan.read(): output = output + self._read_wrapper(o)

And with this modification, i have no more error.

As diagnosis, i suspect that the Fortigate 3000 don't send any caracter to the stream

Commiting firewall rule failed because of parameters order

Hello there,
I've faced this issue and waste a bit of time to debug it but I've finally hunt it !

I use this bit of code to add new firewall rules :

	d.load_config('firewall policy')
	newRule05 = FortiConfig(config_type='edit', name='0')
	newRule05.set_param('srcintf', subnetDescription)
	newRule05.set_param('dstintf', firewallIntercoZone[0])
	newRule05.set_param('srcaddr', 'all')
	newRule05.set_param('dstaddr', fwObjectGrpSrvTransverses)
	newRule05.set_param('action', 'accept')
	newRule05.set_param('status', 'enable')
	newRule05.set_param('service', fwObjectGrpServicesTransverses)
	newRule05.set_param('schedule', 'always')
	d.candidate_config['firewall policy']['0'] = newRule05
	d.commit()

The commit failed intermittently with this generic error :

    raise exceptions.FailedCommit(wrong_commands)
pyFG.exceptions.FailedCommit: [('-3', 'set dstaddr grp-srv-transverses')]

Sometimes it works without error and before this bit of code, the script commits dozens of other rules. The problem is :

• in this rule, the firewall object named grp-srv-transverses is a address group full of addresses attached to one interface
• if you don't specify the dstintf parameter first, you can't specify the dstaddr parameter with this kind of group attached to an interface
• pyFG generates the config section in random order. See the 2 examples beyond for the same rule :

        edit 0
          set action accept
          set schedule always
          set status enable
          set srcaddr all
          set service grp-services-transverses
          set srcintf PP-SUMIT-BD
          set dstintf Interco-TechRH-HProd
          set dstaddr grp-srv-transverses
        next

AND :

        edit 0
          set srcaddr all
          set status enable
          set service grp-services-transverses
          set dstaddr grp-srv-transverses
          set action accept
          set schedule always
          set srcintf PP-SUMIT-BD
          set dstintf Interco-TechRH-HProd
        next

The generation of the rule set should be in a sequential order to avoid this bug.

Regards.

"show full-configuration" and "show" commands return timeout exceptions

Hardware: FortiGate 40C

This code is returning a timeout since it take a long time to retrieve all the configuration :

>>> from pyFG import FortiOS, FortiConfig
>>> d = FortiOS(hostname="10.76.0.252", username="admin", password="***********")
>>> d.open()
>>> d.execute_command('show full-configuration')
Executing commands:
 show full-configuration
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pyFG/fortios.py", line 152, in execute_command
    for e in error_chan.read():
  File "/usr/local/lib/python2.7/dist-packages/paramiko/file.py", line 192, in read
    new_data = self._read(self._DEFAULT_BUFSIZE)
  File "/usr/local/lib/python2.7/dist-packages/paramiko/channel.py", line 1305, in _read
    return self.channel.recv_stderr(size)
  File "/usr/local/lib/python2.7/dist-packages/paramiko/channel.py", line 715, in recv_stderr
    raise socket.timeout()
socket.timeout

In my tests (local network) it takes about :

• 7 minutes to retrieve show full-configuration command result,
• 10/15 seconds to retrieve show command result.

Moving interface between vdoms

Hi
Tried to move Interfaces from vdom to another, when static router still on the Interface, no all nessessay steps are build.