austind/iosfw

Automatic Cisco IOS firmware upgrades

iosfw

Automatic Cisco IOS firmware upgrades

Requires:

• Python 3.6+
• netmiko
• napalm
• tqdm

Overview

Automates the entire upgrade process:

• Determines correct upgrade image for each platform
• Transfer the new image
• Verify image integrity
• Extract archive and install
• Optionally remove old image(s)
• Set boot parameters
• Schedule reload

Auto-detects best upgrade method available:

• archive download-sw
• software install
• request platform software package install
• If those fail, plain copy followed by set boot ...

Supported platforms:

• Catalyst 3550
• Catalyst 3560
• Catalyst 3560-X
• Catalyst 3750
• Catalyst 3750-X
• Catalyst 2960-S
• Catalyst 2960-X
• Catalyst 3650
• Catalyst 3850
• ISR 2921
• ISR 4331
• C892FSP
• ME3400

Experimentally supported platforms:

• ASR920
• Catalyst 9k series

Currently unsupported platforms:

• Nexus 3k/9k series

NOTE: Use at your own risk. It works well in my environment, but serious bugs are possible. Test thoroughly in a lab environment, and see known issues below.

Usage

Preparation

1. Rename config/config.yaml.example to config.yaml, and review config/images.yaml, matching both to your requirements. Defaults are sane, but don't take any chances :)
2. Copy your IOS images defined in images.yaml to the src_image_path defined in config.yaml.

Note: Pay special attention if you have devices of the same model, but need different IOS images (e.g., ipbase vs ipservices). In that case, define both images in images.yaml and add the same model to their respective models lists. Then, change match_feature_set to true in config.yaml.

Interactive Example

>>> from iosfw import iosfw
>>> device = iosfw('ios-sw-1')
>>> device.open()
Username [austindcc]:
Password:
Enable secret:
Opening connection to ios-sw-1...
Connected to ios-sw-1 (WS-C3560X-48P) as austind via ssh
Running version: 12.2(55)SE8
Upgrade version: 15.2(4)E8
Upgrade status: NEEDS UPGRADE
>>> device.upgrade()
Starting upgrade on ios-sw-1 at 14:34:09 06/13/19...
Checking free space...
Found enough free space!
Installing new firmware...
NOTE: No status updates possible during install, which may take 10 minutes or longer.
Install successful!
Removing running image...
Deleting flash:/c3560e-universalk9-mz.122-55.SE8...
Running image deleted.
Scheduling reload...
Reload scheduled for 00:00:00 PDT Fri Jun 14 2019 (9 hours and 16 minutes away)
Upgrade on ios-sw-1 completed at 14:43:32 06/13/19
Total time elapsed: 0:09:23.224298

Automated Example

See example/batch_example.py

Known issues

• Only supports BUNDLE mode on cat9k. As of 2021-06-16, INSTALL mode reloads the device immediately, with no option to delay reload for image activation. This is not only inconvenient, but also complicates iosfw's state awareness.
• As of 0.9.0, SCP image transfer directly from iosfw no longer works. I recommend setting up an FTP server on a separate host and setting config.yaml accordingly.
• Catalyst 3k series (3650 and 3850) with IOS running in BUNDLE mode (booted directly to the .bin file), will not succeed in upgrading with request platform software package install. Upgrading them requires a different manual process that is not yet implemented:
  • Remove existing IOS packages: del /force flash:/cat*.pkg
  • Remove existing packages.conf: del /force flash:/packages.conf
  • Remove boot variables: no boot system in config mode
  • Copy upgrade image: copy <source> flash:
  • Install upgrade image: request platform software package expand switch all file flash:/<file>
  • Set boot variable: boot system flash:/<file>
  • Schedule reload: reload at 00:00
• Currently, iosfw does not check to ensure transfer_source is reachable. If not reachable, the install command will fail, but not timeout for more than 30 minutes. Most commonly, transfer_source may not be reachable due to sending the requests out the incorrect interface. You can specify the source interface for TFTP and FTP transfers with ip (ftp|tftp) source-interface <iface> in config mode.
• When using SSH proxy, iosfw throws a ProcessLookupError on exit. I have not found a way to catch or suppress this.
• When remove_old_images is set to always on platforms using request software... install method, iosfw may incorrectly remove the newly installed image files. Net result is no change to system. Workaround: use remove_old_images = as_needed.
• Totally untested on stacks.

Wishlist

• Nornir integration
• Fix native SCP image transfer option (broken as of 0.9.0)
• Accept a pre-existing napalm connection object
• Verify reachability of transfer_source, attempting fix as needed
• More consistent debug output
• N3K/N9K support

Contributions welcome.

Changelog

See CHANGELOG.md

Notes

• Expect most upgrades to take 8-10 minutes per device, with one notable exception: Catalyst 3750-X took no less than 40 minutes in testing.
• Expect devices to take between 10 and 30 minutes to come back after reload, especially if upgrading trains or major versions, due to microcode updates.
• The automated install commands (archive download-sw and request platform software package install) download the upgrade package twice, for reasons I did not determine.
• FTP and HTTP seem to be the fastest transfer methods. Even then, the download appears constrained by platform CPU resources, averaging about 4Mbps in most tests, while some newer platforms achieved 20Mbps.
• The iosfw class exposes all of NAPALM's config parameters, and stores the NAPALM session under self.napalm, so you can use all of NAPALM's features easily.
• Same goes for netmiko - stored as self.device - so you can send arbitrary commands with iosfw.device.send_command('my arbitrary command')

iosfw's Issues

reload error

Hi, i'm getting the following error when I try to run it:

ValueError: Tried to schedule reload with `reload at 00:00`, but check_reload() failed. Output:
It all works fine, it even schedules the reload but the error is always raised.

pass `reload_at` as argument

Is there an existing way to pass more config elements as an argument, which would override what it sees in config.yaml?

An example of this would be a script that takes a few arguments, one of which would pass to reload_at manually. Use case is a device-upgrader wrapper script that you send the hostname and reload_at time (in cisco mmm / hhh:mm format natively) which would permit a single script to upgrade a device and have a custom reload time, which is driven by customer.

I eyeballed iosfw.py but didn't see a native way to do this. could be wrong. Was mainly looking at the # Set up connection section which has a few if's.

Hide output of config settings

How does one hide the output of the python dictionary of the config file? I was doing testing via cli, but have moved to integrating it in to a script using batch_example.py.

When I run it, I always get the full output of config.yaml dumped to console, as well as

I've tried adjusting console_log_level from info to critical, and no luck in between. I've also changed log_method from tee to file, also with no changes at all.

3850 Improve speed

I have found out that when doing http transfer on 3850s (probably 3650s), it takes forever (denali and up) - mine takes almost 6hours.
Increasing tcp windows size, helps tremendously:

ip tcp window-size 1073741823
ip tcp path-mtu-discovery

I would try a few window sizes, this is the maximum window size allowed (maybe try/expect and decrease it if there's an IO error - I tried it on a 2960X and it failed a few times, I'll need to do further testing)

Below is an example of the file transfer across the atlantic, only 291.601 seconds, before this it was 5+ hours.

-CSWT02#copy http://10.95.238.50/cat3k_caa-universalk9.16.08.01a.SPA.bin flash: Destination filename [cat3k_caa-universalk9.16.08.01a.SPA.bin]? Accessing http://10.95.238.50/cat3k_caa-universalk9.16.08.01a.SPA.bin... Loading http://10.95.238.50/cat3k_caa-universalk9.16.08.01a.SPA.bin !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 425928073 bytes copied in 291.601 secs (1460654 bytes/sec)

iosfw 0.9.6 ProcessLookupError

Testing iosfw 0.9.6, I having errors. Nothing is logged to file on disk, only throwing:

$ python3
Python 3.6.8 (default, Apr  2 2020, 13:34:55) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from iosfw import iosfw
>>> device = iosfw('dhcp-buildroom-7')
Exception ignored in: <bound method iosfw.__del__ of <iosfw.iosfw.iosfw object at 0x7f7eabaf17b8>>
Traceback (most recent call last):
  File "/opt/scripts/iosfw/iosfw/iosfw.py", line 227, in __del__
    except OSError.ProcessLookupError:
AttributeError: type object 'OSError' has no attribute 'ProcessLookupError'
>>> 
>>> 
KeyboardInterrupt

That error is in a function that's attempting to work around an ssh proxy, which we are not using. If it matters, I have these versions of related things installed:

napalm (3.0.1)
netmiko (3.1.1)
PyYAML (5.3.1)

If I switch back to the dev branch, which I was testing before a few recent bugfixes, it works fine.

ASR920 IOS XE filename _get_version_from_file() issues

I am testing iosfw on an ASR920 with this filename:

asr920-universalk9_npe.16.09.05f.SPA.bin

It fails with:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/falz/testscripts/iosfw/iosfw/iosfw.py", line 149, in __init__
    self.firmware_installed = self.check_firmware_installed()
  File "/home/falz/testscripts/iosfw/iosfw/iosfw.py", line 757, in check_firmware_installed
    if version in files:
TypeError: 'in <string>' requires string as left operand, not NoneType

Cisco appears to have thrown a wrench in to _get_version_from_file() in that the 'f' character is inserted in this IOS version. The problematic line in iosfw.py is:

pattern = r"(\d+\.\d+\.\d+)\.SPA"

browsing filenames on downloads for this device, some releases have a letter there, some do not:
https://software.cisco.com/download/home/286288711/type/282046477/release/Fuji-16.9.5f

I see an example of "f" in my case, "a" in another case.

I tested changing from pattern

(\d+\.\d+\.\d+)\.SPA

to

(\d+\.\d+\.\d+)([a-z]|)\.SPA

And it seemed to work (on regexpal.com), but there's something off about the pipe followed by nothing.. I'm not sure the most efficient way to match a string OR nothing.

Another potential issue after this is that this will return a string instead of an integer, which I suspect may break something further in the code.

Endless recursion on trying NTP Update

Hi,

in iosfw.py:825 there is a call for schedule_reload() within the definition. If the NTP sync doesn't happen the function will call itself until the end of days.

def schedule_reload(...)
[...]
 return self.schedule_reload()
[...]

Thanks for the neat piece of code

Moe

reload_at config variable ignored

Hi there,

Doing some testing of this on Cisco ME3400 devices as a method to mass upgrade.

We're noticing that the 'reload_at' setting we have in config.yaml is being ignored. We changed the default 00:00 to 06:00, but it still uses 00:00. Other things from our config.yaml are being accepted.

The reload_* portion of config.yaml looks like this (default commented out):

reload_in: false
#reload_in: 10
reload_at: '06:00'

I've tested this on a handful of devices, all me3400, always with the same results. Config syntax above perhaps?

ASR920 ROMMON upgrade support

Splitting off from issue 7 is the ability to upgrade ROMMON, in this case for ASR920. Consider this a feature request.

ROMMON on asr920 is a separate package to install. Cisco's docs are poor, but imply that it should be upgraded, so it seems that it may fit with this project. If not, feel free to WONTFIX.

Checking ROMMON version
Confusingly, checking ROMMON version is available in at least three commands. It's "15.6(10r)S" in the output below:

r-asr920-lab#show rom-monitor r0
System Bootstrap, Version 15.6(10r)S, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2016 by cisco Systems, Inc.

r-asr920-lab#show rom-monitor rp active
System Bootstrap, Version 15.6(10r)S, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2016 by cisco Systems, Inc.

r-asr920-lab#show platform
Chassis type: ASR-920-4SZ-A
 
Slot      Type                State                 Insert time (ago)
--------- ------------------- --------------------- -----------------
 0/0      2xGE-4x10GE-FIXED   ok                    5d14h        
R0        ASR-920-4SZ-A       ok, active            5d14h        
F0                            ok, active            5d14h        
P0        ASR920-PSU0         ps, fail              never        
P1        ASR920-PSU1         ok                    never        
P2        ASR920-FAN          ok                    never        
 
Slot      CPLD Version        Firmware Version                       
--------- ------------------- ---------------------------------------
R0        19052043            15.6(10r)S
F0        19052043            15.6(10r)S

Upgrading ROMMON
Upgrade procedure is to place the new rommon file on disk, and then run an upgrade command.

copy http://<url>/asr920_15_6_32r_s_rommon.pkg bootflash:
upgrade rom-monitor filename bootflash:asr920_15_6_32r_s_rommon.pkg all

This command takes about a minute to run, then requests a reload. Note that we have done this AND an IOS upgrade in the same reload and it seems fine, so no separate reload for this seems to be required.

r-asr920-lab#upgrade rom-monitor filename bootflash:asr920_15_6_32r_s_rommon.pkg all
 
Upgrade rom-monitor on Route-Processor 0
 
Target copying rom-monitor image file
unlock rommon partition
Checking upgrade image...
1966080+0 records in
3840+0 records out
1966080 bytes (2.0 MB, 1.9 MiB) copied, 4.4175 s, 445 kB/s
Upgrade image MD5 signature is 40966c321c22ab1671ab5617c4ce3b79
Burning upgrade partition...
1966080+0 records in
1966080+0 records out
1966080 bytes (2.0 MB, 1.9 MiB) copied, 29.0814 s, 67.6 kB/s
Checking upgrade partition...
1966080+0 records in
1966080+0 records out
1966080 bytes (2.0 MB, 1.9 MiB) copied, 23.0408 s, 85.3 kB/s
Upgrade flash partition MD5 signature is 40966c321c22ab1671ab5617c4ce3b79
ROMMON upgrade complete.
To make the new ROMMON permanent, you must restart the RP.
 
r-asr920-lab#

For the sake of completeness or for any others reading this, the serial console will show this type of output while booting, which indicates success. This doesn't add much time to the already excruciatingly long boot time of this platform.

System Bootstrap, Version 15.6(10r)S, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2016 by cisco Systems, Inc.
Compiled Thu 24-Mar-16 15:38 by alnguyen
Boot ROM1
Last reset cause: RSP-Board
Rommon upgrade requested
Flash upgrade reset 1 in progress
.......
 
System Bootstrap, Version 15.6(32r)S, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2018 by cisco Systems, Inc.
Compiled Thu 30-Aug-18 06:23 by pallavik
*Upgrade in progress* Boot ROM0
Last reset cause: BootRomUpgrade
 

Build pip package

When compiling from source, python 10 complains loudly that you're not installing via pip.

request to add whichever files let you build a pip module, and while there maybe submit officially?

Transport - Telnet?

Hi there!

I've been playing around with this hoping to use it to upgrade a bunch of switches that currently only support telnet. Is there a configuration flag to to use telnet instead of SSH? It looks like napalm has support for it but I'm struggling to find out exactly what configuration entry to change.

Thank you kindly for any pointers.

Working on Cisco device?

Hi!

Will this work on following:
Cisco 4506
Cisco 9007
Cisco ASR 9001
Cisco ASR 9401

or maybe can recommend something. Would like a “version” check and upgrade system.