nautobot / nautobot-plugin-ssot-ipfabric Goto Github PK
View Code? Open in Web Editor NEWNautobot SSoT IP Fabric
Home Page: https://nautobot.github.io/nautobot-plugin-ssot-ipfabric/
License: Other
Nautobot SSoT IP Fabric
Home Page: https://nautobot.github.io/nautobot-plugin-ssot-ipfabric/
License: Other
Ensure all adapters have create, update and delete.
When performing the sync between IP Fabric and Nautobot, I have an error, and the sync fails:
An exception occurred: DataError: value too long for type character varying(50)
Sync failure. Doing a few tests, I've added in a recent snapshot Azure devices, which have a very long SN. By removing those devices, I can sync without any problem. It seems that the limit of 50 is too small and causes this issue. Here is a couple of example of the SN we have for Azure devices:
Hostname | Site | Unique serial number |
---|---|---|
VNetGw.Express.WestEurope | AZURE | /subscriptions/7a329198-ceb1-4e25-xxxx-e98855c123456/resourceGroups/LAB-Static/providers/Microsoft.Network/virtualNetworkGateways/VNetGw.Express.WestEurope |
nat-Gateway | AZURE | /subscriptions/7a329198-ceb1-4e25-xxxx-e98855c123456/resourceGroups/LAB-Static/providers/Microsoft.Network/natGateways/nat-Gateway |
Here are the logs of the error:
Traceback (most recent call last):
File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/query.py", line 573, in get_or_create
return self.get(**kwargs), False
File "/opt/nautobot/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
return qs._no_monkey.get(qs, *args, **kwargs)
File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/query.py", line 429, in get
raise self.model.DoesNotExist(
nautobot.dcim.models.devices.Device.DoesNotExist: Device matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/nautobot/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.StringDataRightTruncation: value too long for type character varying(50)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot/jobs/base.py", line 326, in run
self.sync_data()
File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/jobs.py", line 185, in sync_data
dest.sync_from(ipfabric_source)
File "/opt/nautobot/lib/python3.8/site-packages/diffsync/__init__.py", line 525, in sync_from
result = syncer.perform_sync()
File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 326, in perform_sync
changed |= self.sync_diff_element(element)
File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 390, in sync_diff_element
changed |= self.sync_diff_element(child, parent_model=dst_model)
File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 366, in sync_diff_element
changed, modified_model = self.sync_model(src_model=src_model, dst_model=dst_model, ids=ids, attrs=attrs)
File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 415, in sync_model
dst_model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs)
File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/diffsync/diffsync_models.py", line 191, in create
new_device, _ = NautobotDevice.objects.get_or_create(
File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/query.py", line 576, in get_or_create
return self._create_object_from_params(kwargs, params)
File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/query.py", line 610, in _create_object_from_params
obj = self.create(**params)
File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/query.py", line 447, in create
obj.save(force_insert=True, using=self.db)
File "/opt/nautobot/lib/python3.8/site-packages/nautobot/dcim/models/devices.py", line 735, in save
super().save(*args, **kwargs)
File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/base.py", line 753, in save
self.save_base(using=using, force_insert=force_insert,
File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/base.py", line 790, in save_base
updated = self._save_table(
File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/base.py", line 895, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/base.py", line 933, in _do_insert
return manager._insert(
File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/query.py", line 1254, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1397, in execute_sql
cursor.execute(sql, params)
File "/opt/nautobot/lib/python3.8/site-packages/cacheops/transaction.py", line 93, in execute
result = self._no_monkey.execute(self, sql, params)
File "/opt/nautobot/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/opt/nautobot/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/opt/nautobot/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/opt/nautobot/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/nautobot/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.DataError: value too long for type character varying(50)
Job debug logging shows in output even if debug isn't checked. Logs show under 'default'
Be able to define multiple sources to sync from and select them when running the Job
Allow user to select which snapshot to target from IPFabric
Currently we have defaulted the following:
DEFAULT_DEVICE_ROLE = "Network Device"
DEFAULT_DEVICE_STATUS = "Active"
Allow for these values to be configurable through settings from the user, as there is no way to pull this from ipfabric
nautobot==1.5.7
nautobot-capacity-metrics==2.0.0
nautobot-chatops==1.10.0
nautobot-chatops-ipfabric==3.0.1
nautobot-ssot==1.2.0
nautobot-ssot-ipfabric==2.0.0
To start SSOT
Need to debug more but ran out of time.
*Sync failed. Here is the link to your job: https://nautobot-demo.ipf.cx/plugins/ssot/history/330fa397-f1d8-4068-af93-a1752549c0e6/*
File "/opt/nautobot/lib/python3.9/site-packages/nautobot_ssot/jobs/base.py", line 332, in run
self.sync_data()
File "/opt/nautobot/lib/python3.9/site-packages/nautobot_ssot_ipfabric/jobs.py", line 225, in sync_data
self.client.snapshot_id = self.kwargs["snapshot"]
KeyError: 'snapshot'
/ipfabric ssot-sync-to-nautobot True True True
Interfaces always appear as created during diffsync reports. This should not be the case, once they are created, only updates should happen and diff sync should not be reporting on them, unless updated or deleted.
Brings me to a filtered table for the history of a job and a specific action
URL: /plugins/ssot/logs/?overview=15cbe6d1-a0c3-4379-a17a-d7c2f2f4e16c&action=create
Invalid filters were specified:
overview
Unknown filter field
We should try and get auto docs works to document API code
Document what gets synced over for each model, etc.
Create IP Fabric DiffSync Adapter
Make a call to part numbers in IP Fabric to map to vendor (manufacturer), model (device type), potentially platform? May require mapping? Serial number
Interface objects populated from IPFabric seems to hit an error with device imports. When there is no description on the device, the plug-in should send an empty string for the interface, as per the second example below.
interface_obj.description = fields.get ("description")
interface_obj.description = fields.get ("description", โโ)
Get rid of travis file and move pipelines to GH Actions
Integrate chatops
Add Device model to sync from IP Fabric
Before testing any utilities, switch them over to get_or_create as appropriate
As part of the topology rendering from IP Fabric, we display the a interactive image on sites which use the "facility" field to populate the necessary link.
The request is to not hijack this field for a plugin and use custom_field_data
and call a simple .update("topology_id") to store this value on an individual site basis while running the sync job
Nautobot starts up when plugin is added with steps described in README
Nautobot worker crashes on startup with traceback below
ipfabric_worker_1 exited with code 1
nautobot_1 | Traceback (most recent call last):
nautobot_1 | File "/usr/local/bin/nautobot-server", line 8, in <module>
nautobot_1 | sys.exit(main())
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/nautobot/core/cli.py", line 54, in main
nautobot_1 | run_app(
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/nautobot/core/runner/runner.py", line 266, in run_app
nautobot_1 | management.execute_from_command_line([runner_name, command] + command_args)
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
nautobot_1 | utility.execute()
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
nautobot_1 | django.setup()
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
nautobot_1 | apps.populate(settings.INSTALLED_APPS)
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 122, in populate
nautobot_1 | app_config.ready()
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/nautobot_ssot_ipfabric/__init__.py", line 37, in ready
nautobot_1 | super().ready()
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/nautobot/extras/plugins/__init__.py", line 144, in ready
nautobot_1 | jobs = import_object(f"{self.__module__}.{self.jobs}")
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/nautobot/extras/plugins/utils.py", line 45, in import_object
nautobot_1 | spec.loader.exec_module(module)
nautobot_1 | File "<frozen importlib._bootstrap_external>", line 843, in exec_module
nautobot_1 | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/nautobot_ssot_ipfabric/jobs.py", line 77, in <module>
nautobot_1 | CLIENT = IPFClient(
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/ipfabric/client.py", line 45, in __init__
nautobot_1 | super().__init__(base_url, token, snapshot_id, username, password, **kwargs)
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/ipfabric/api.py", line 71, in __init__
nautobot_1 | self.os_version = self.fetch_os_version()
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/ipfabric/api.py", line 97, in fetch_os_version
nautobot_1 | res = self.get(url="os/version")
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1039, in get
nautobot_1 | return self.request(
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 815, in request
nautobot_1 | return self.send(request, auth=auth, follow_redirects=follow_redirects)
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 902, in send
nautobot_1 | response = self._send_handling_auth(
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 930, in _send_handling_auth
nautobot_1 | response = self._send_handling_redirects(
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 967, in _send_handling_redirects
nautobot_1 | response = self._send_single_request(request)
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1003, in _send_single_request
nautobot_1 | response = transport.handle_request(request)
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 218, in handle_request
nautobot_1 | resp = self._pool.handle_request(req)
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 253, in handle_request
nautobot_1 | raise exc
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 237, in handle_request
nautobot_1 | response = connection.handle_request(request)
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 86, in handle_request
nautobot_1 | raise exc
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 63, in handle_request
nautobot_1 | stream = self._connect(request)
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 111, in _connect
nautobot_1 | stream = self._network_backend.connect_tcp(**kwargs)
nautobot_1 | File "/usr/local/lib/python3.8/site-packages/httpcore/backends/sync.py", line 86, in connect_tcp
nautobot_1 | sock = socket.create_connection(
nautobot_1 | File "/usr/local/lib/python3.8/socket.py", line 793, in create_connection
nautobot_1 | sock.settimeout(timeout)
nautobot_1 | TypeError: an integer is required (got type str)
To test if this was really the issue I wrapped the ipfabric_timeout
value in the plugins config in the int() method and nautobot started as expected.
"nautobot_ssot_ipfabric": {
"ipfabric_api_token": os.environ.get("IPFABRIC_API_TOKEN"),
"ipfabric_host": os.environ.get("IPFABRIC_HOST"),
"nautobot_host": os.environ.get("NAUTOBOT_HOST"),
"ipfabric_ssl_verify": os.environ.get("IPFABRIC_SSL_VERIFY"),
"ipfabric_timeout": int(os.environ.get("IPFABRIC_TIMEOUT", 15)),
},
Make sure UI is showing ability to sync from IP Fabric
Ability to use the chatops plugin to start syncing and reporting back status potentially? Raw dump of data we get from the job.
A connection to the IPFabric server should be established only when triggered an action and not on module import.
When importing a module, this piece of code is executed:
nautobot-plugin-ssot-ipfabric/nautobot_ssot_ipfabric/jobs.py
Lines 76 to 86 in 1c76fec
If an API token or a host is not set, we get an error:
Traceback (most recent call last):
File "/usr/local/bin/nautobot-server", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/nautobot/core/cli.py", line 54, in main
run_app(
File "/usr/local/lib/python3.9/site-packages/nautobot/core/runner/runner.py", line 266, in run_app
management.execute_from_command_line([runner_name, command] + command_args)
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 395, in execute
django.setup()
File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 122, in populate
app_config.ready()
File "/usr/local/lib/python3.9/site-packages/nautobot_ssot_ipfabric/__init__.py", line 37, in ready
super().ready()
File "/usr/local/lib/python3.9/site-packages/nautobot/extras/plugins/__init__.py", line 143, in ready
jobs = import_object(f"{self.__module__}.{self.jobs}")
File "/usr/local/lib/python3.9/site-packages/nautobot/extras/plugins/utils.py", line 45, in import_object
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/usr/local/lib/python3.9/site-packages/nautobot_ssot_ipfabric/jobs.py", line 77, in <module>
CLIENT = IPFClient(
File "/usr/local/lib/python3.9/site-packages/ipfabric/client.py", line 45, in __init__
super().__init__(base_url, token, snapshot_id, username, password, **kwargs)
File "/usr/local/lib/python3.9/site-packages/ipfabric/api.py", line 64, in __init__
raise RuntimeError("IP Fabric Token or Username/Password not provided.")
RuntimeError: IP Fabric Token or Username/Password not provided.
I think we shouldn't add any piece of code that establishes a connection outside a function or a class in the python file, as this is executed when importing the module.
0.10.0
.We need to include the chatops plugin to re-use the client and maybe other magical things
The site diagram iframe
is not longer available on newer ipfabric versions
Add ability to sync interfaces from IP Fabric
Can install ipfabric-chatops and ipfabric-ssot on the same host
Dependency issue with python-ipfabric library
IP Fabric classifies devices into different "Types" (column devType
) which can be used as a Nautobot Device Role
. Currently all devices are sync'ed using the default:
DEFAULT_DEVICE_ROLE = CONFIG.get("default_device_role", "Network Device")
DEFAULT_DEVICE_ROLE_COLOR = CONFIG.get("default_device_role_color", "ff0000")
nautobot_config.py
that enables syncing the IP Fabric Device Type to Nautobot Device Role.
FQDN/api/v#.#/tables/inventory/devices
FQDN/inventory/devices
devType
> devices = ipf.inventory.devices.all()
> dev_type = {d['devType'] for d in devices}
> print(dev_type)
{'switch', 'fw', 'waas', 'wlc', 'lb', 'l3switch', 'router'}
Network Device
is too generic."List of Device Types in IP Fabric v6.2.1 is shown below. Not all are used for Network Device Inventory
purposes. For instance host
is used for IPF discovered hosts.
[
'aciLeaf',
'aciSpine',
'ap',
'apic',
'cloudInstance',
'cloudInternetGw',
'cloudLoadBalancer',
'cloudNatGw',
'cloudRouter',
'cloudTransitHub',
'cloudVpnGw',
'fex',
'fw',
'host',
'l3switch',
'lb',
'nx7000',
'phone',
'router',
'securityManagement',
'switch',
'unknown',
'vgw',
'waas',
'wlc',
]
Add Site model to sync site data from IPFabric to Nautobot
Add ability to sync VLAN information from IP Fabric
Some sort of presentation slide to highlight how we structured and put together a winning plan from the start
SSOT errors when Safe Delete Mode is not enabled.
File "/opt/nautobot/lib/python3.9/site-packages/nautobot_ssot/jobs/base.py", line 332, in run
self.sync_data()
File "/opt/nautobot/lib/python3.9/site-packages/nautobot_ssot_ipfabric/jobs.py", line 277, in sync_data
dest.sync_from(ipfabric_source)
File "/opt/nautobot/lib/python3.9/site-packages/diffsync/__init__.py", line 526, in sync_from
self.sync_complete(source, diff, flags, syncer.base_logger)
File "/opt/nautobot/lib/python3.9/site-packages/nautobot_ssot_ipfabric/diffsync/adapter_nautobot.py", line 72, in sync_complete
nautobot_object.delete()
File "/opt/nautobot/lib/python3.9/site-packages/django/db/models/base.py", line 960, in delete
assert self.pk is not None, (
AssertionError: Interface object can't be deleted because its id attribute is set to None.
Normal completion at the end of the sync
An exception occurred: ValidationError: {'__all__': ['assigned_object_type and assigned_object_id must either both be null or both be non-null']}
Traceback (most recent call last): File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot/jobs/base.py", line 326, in run self.sync_data() File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/jobs.py", line 185, in sync_data dest.sync_from(ipfabric_source) File "/opt/nautobot/lib/python3.8/site-packages/diffsync/__init__.py", line 525, in sync_from result = syncer.perform_sync() File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 326, in perform_sync changed |= self.sync_diff_element(element) File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 390, in sync_diff_element changed |= self.sync_diff_element(child, parent_model=dst_model) File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 390, in sync_diff_element changed |= self.sync_diff_element(child, parent_model=dst_model) File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 366, in sync_diff_element changed, modified_model = self.sync_model(src_model=src_model, dst_model=dst_model, ids=ids, attrs=attrs) File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 415, in sync_model dst_model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs) File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/diffsync/diffsync_models.py", line 306, in create ip_address_obj = tonb_nbutils.create_ip( File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/utilities/nbutils.py", line 143, in create_ip tag_object(nautobot_object=ip_obj, custom_field="ssot-synced-from-ipfabric") File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/utilities/nbutils.py", line 253, in tag_object _tag_object(nautobot_object) File "/opt/nautobot/lib/python3.8/site-packages/nautobot_ssot_ipfabric/utilities/nbutils.py", line 251, in _tag_object nautobot_object.validated_save() File "/opt/nautobot/lib/python3.8/site-packages/nautobot/core/models/__init__.py", line 51, in validated_save self.full_clean() File "/opt/nautobot/lib/python3.8/site-packages/django/db/models/base.py", line 1251, in full_clean raise ValidationError(errors) django.core.exceptions.ValidationError: {'__all__': ['assigned_object_type and assigned_object_id must either both be null or both be non-null']}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.