nautobot/nautobot-app-netbox-importer

Nautobot plugin to simplify data migration from NetBox

Nautobot

Nautobot is a Network Source of Truth and Network Automation Platform built as a web application atop the Django Python framework with a PostgreSQL or MySQL database.

Key Use Cases

1. Flexible Source of Truth for Networking - Nautobot core data models are used to define the intended state of network infrastructure enabling it as a Source of Truth. While a baseline set of models are provided (such as IP networks and addresses, devices and racks, circuits and cable, etc.) it is Nautobot's goal to offer maximum data model flexibility. This is enabled through features such as user-defined relationships, custom fields on any model, and data validation that permits users to codify everything from naming standards to having automated tests run before data can be populated into Nautobot.

2. Extensible Data Platform for Automation - Nautobot has a rich feature set to seamlessly integrate with network automation solutions. Nautobot offers GraphQL and native Git integration along with REST APIs and webhooks. Git integration dynamically loads YAML data files as Nautobot config contexts. Nautobot also has an evolving plugin system that enables users to create custom models, APIs, and UI elements. The plugin system is also used to unify and aggregate disparate data sources creating a Single Source of Truth to streamline data management for network automation.

3. Platform for Network Automation Apps - The Nautobot plugin system enables users to create Network Automation Apps. Apps can be as lightweight or robust as needed based on user needs. Using Nautobot for creating custom applications saves up to 70% development time by re-using features such as authentication, permissions, webhooks, GraphQL, change logging, etc. all while having access to the data already stored in Nautobot. Some production ready applications include:

The complete documentation for Nautobot can be found at Read the Docs.

Questions? Comments? Start by perusing our GitHub discussions for the topic you have in mind, or join the #nautobot channel on Network to Code's Slack community!

Build Status

Branch	Status
main
develop
next

Screenshots

---

---

---

---

Installation

Please see the documentation for instructions on installing Nautobot.

Application Stack

Below is a simplified overview of the Nautobot application stack for reference:

Plugins and Extensibility

Nautobot offers the ability to customize your setup to better align with your direct business needs. It does so through the use of various plugins that have been developed for network automation, and are designed to be used in environments where needed.

There are many plugins available within the Nautobot Apps ecosystem. The below screenshots are an example of some popular ones that are currently available.

Plugin Screenshots

Golden Config Plugin

ChatOps Plugin

Device Lifecycle Management Plugin

Providing Feedback

The best platform for general feedback, assistance, and other discussion is our GitHub discussions. To report a bug or request a specific feature, please open a GitHub issue using the appropriate template.

If you are interested in contributing to the development of Nautobot, please read our contributing guide prior to beginning any work.

Related projects

Please check out the GitHub nautobot topic for a list of relevant community projects.

Notices

Nautobot was initially developed as a fork of NetBox (v2.10.4). NetBox was originally developed by Jeremy Stretch at DigitalOcean and the NetBox Community.

nautobot-app-netbox-importer's Issues

Nautobot 1.5.3 makes test fail

Environment

• Python version: 3.8
• Nautobot version: 1.5.3
• nautobot-netbox-importer version: 1.4.2

Expected Behavior

Unittests pass

Observed Behavior

System check identified no issues (0 silenced).
..........
======================================================================
FAIL: test_imported_data_correctness (nautobot_netbox_importer.tests.test_import.TestImport) (content_type='extras.customfield', ids={'name': 'Custom Text Field A'})
Iterate over all imported data and check its correctness.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/source/nautobot_netbox_importer/tests/test_import.py", line 104, in test_imported_data_correctness
    self.assertEqual(expected_value, actual_value, f"key '{key}' on {model} '{record}' is incorrect")
AssertionError: '' != 'Custom Text Field A'
+ Custom Text Field A : key 'label' on customfield 'Custom Text Field A' is incorrect

======================================================================
FAIL: test_imported_data_correctness (nautobot_netbox_importer.tests.test_import_objectchange.TestImportObjectChange) (content_type='extras.customfield', ids={'name': 'Custom Text Field A'})
Iterate over all imported data and check its correctness.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/source/nautobot_netbox_importer/tests/test_import.py", line 104, in test_imported_data_correctness
    self.assertEqual(expected_value, actual_value, f"key '{key}' on {model} '{record}' is incorrect")
AssertionError: '' != 'Custom Text Field A'
+ Custom Text Field A : key 'label' on customfield 'Custom Text Field A' is incorrect

----------------------------------------------------------------------
Ran 12 tests in 37.242s

Steps to Reproduce

1. invoke unittest

Plugin imports custom fields but not their values

Environment

• Python version: 3.6.8
• Nautobot version: 1.0.1
• nautobot-netbox-importer version: 1.2.1

Expected Behavior

Migrate all custom fields and their values

Observed Behavior

Custom fields are migrated but their values are not (values are blank or set to defaults).

Steps to Reproduce

1. $ sudo -u postgres psql
2. postgres=# DROP DATABASE nautobot;
3. postgres=# CREATE DATABASE nautobot;
4. postgres=# GRANT ALL PRIVILEGES ON DATABASE nautobot TO nautobot;
5. $ sudo systemctl restart nautobot nautobot-worker
6. $ nautobot-server migrate
7. $ nautobot-server createsuperuser
8. $ nautobot-server collectstatic
9. $ nautobot-server check
10. $ python netbox/manage.py dumpdata --traceback --format=json --exclude admin.logentry --exclude sessions.session --exclude extras.ObjectChange --exclude extras.Script --exclude extras.Report > /tmp/netbox_data.json
11. $ nautobot-server import_netbox_json /tmp/netbox_data.json 2.10.7

Add support for importing NetBox 3.0.x data

With #58 as a pre-requisite, the following changes in NetBox 3.0.x need to be accounted for by the importer:

• Added IPRange model
  • No current equivalent in Nautobot, will need to be ignored/discarded for now
• Added interface/powerport type enums (netbox-community/netbox#7052, netbox-community/netbox@2ba6a6f, netbox-community/netbox@bfb37d6, netbox-community/netbox@854121b, netbox-community/netbox#7486, netbox-community/netbox#7774, netbox-community/netbox#7816, netbox-community/netbox#7915, netbox-community/netbox#7941)
  • Not generally supported in Nautobot at this time, will need to coerce to the closest relevant type.
• Added color field to various port types
  • No current equivalent in Nautobot
• Added "km" and "mi" Cable length units
  • Will need to coerce to Nautobot supported units at present
• Changed Cable.length to a decimal field
  • Will need to coerce back to an integer for Nautobot at present
• Added ImageAttachment support for Circuit and PowerPanel
  • No current equivalent in Nautobot

Document Seed or Root Key for pk deterministic

Environment

• Python version: 3.7
• Nautobot version: Current
• nautobot-netbox-importer version: Current

Expected Behavior

The documentation of the netbox_pk_to_nautobot_pk and that it uses the SECRET_KEY for creating the deterministic pk to pk function

Observed Behavior

Not Documented

Steps to Reproduce

N/A

Add support for importing NetBox 3.3.x data

With #92 as a prerequisite, the following NetBox data model changes may need to be accounted for in the importer. Changes in models already noted in prior issues as not relevant to Nautobot (such as WirelessLAN) are not included below for brevity.

• Change of IPAddress.nat_inside from a OneToOneField to a ForeignKey (allowing multiple nat_outside records) - a similar feature was added in Nautobot 1.3
• Addition of VirtualMachine.site and VirtualMachine.device fields - no Nautobot equivalent
• Change of VirtualMachine.cluster field to nullable - null values are not presently permitted in Nautobot, will need handling
• Addition of Circuit.termination_date field - no Nautobot equivalent
• Addition of RackGroupLocation.status and Cluster.status fields - no Nautobot equivalent
• Support for custom fields and tags on CircuitTermination - Nautobot has supported custom fields here since 1.0 and tags since 1.3.
• Removal of maximum length limits on CustomLinks - will need to be truncated on import to Nautobot
• Addition of ConfigContext.locations field - recall that NetBox's locations are what Nautobot calls rack groups, this is not the same as Nautobot's ConfigContext.locations field. No Nautobot equivalent.
• Addition of Token.allowed_ips and Token.last_used fields - no Nautobot equivalent
• Addition of CustomField.group_name - equivalent to Nautobot CustomField.grouping added in Nautobot 1.5
• Addition of CustomField.ui_visibility - similar to Nautobot CustomField.advanced_ui added in Nautobot 1.3
• Change of DeviceType.u_height and Device.position to support decimal values - will need to be coerced to integers for Nautobot, will need to think about how to handle adjacent devices spanning partial rack units though.
• Addition of Interface.poe_mode and Interface.poe_type - no Nautobot equivalent
• Addition of L2VPN and L2VPNTermination models - no Nautobot equivalent
• Major changes to the Cable, CablePath, and CableTermination models, and associated changes to all cable-able port models - this will likely be difficult to support at this time.
• Additional type enums (netbox-community/netbox#10116, netbox-community/netbox#10554, netbox-community/netbox@1e0b024, etc.)

Error message logged on encountering NAPALM permissions

Environment

• Python version:
• Nautobot version: 1.0.0b3
• nautobot-netbox-importer version: 1.2.0

Expected Behavior

Successful import without error messages

Observed Behavior

15:00:22 error    Expected but did not find an existing Nautobot record
  source: user
  target: permission
  unique_id:
    {'codename': 'napalm_read', 'content_type': <ContentType: dcim | device>}
15:00:22 error    Expected but did not find an existing Nautobot record
  source: user
  target: permission
  unique_id:
    {'codename': 'napalm_write', 'content_type': <ContentType: dcim | device>}

(may be repeated multiple times)

Steps to Reproduce

1. Import a dump from NetBox that has these permission records defined.

Interfaces connected to single-termination circuits have broken path traces/connected_endpoint property

Environment

• Python version: 3.7.3
• Nautobot version: 1.2.2
• nautobot-netbox-importer version: 1.4.0

Expected Behavior

After importing data from NetBox, all imported items, including single-termination circuits (e.g. transits, only one termination exists or is in use), to work and show up correctly

Observed Behavior

The path showed up as 'not reachable'. The 'connected_endpoint' of the interface and circuit termination was also null. The circuit trace functionality was also broken on the interface.

What did not help in resolving this:

• Delete extraneous circuit termination
• Update circuit termination in use
• Update circuit

What resolves the issue:

• Deleting unused side of the circuit termination
• AND
  -> Updating the cable connecting the circuit termination and interface
  -> Deleting and recreating the cable connecting the circuit termination and interface

Steps to Reproduce

1. Create a transit circuit in NetBox and connect it to an Interface of a Device
2. Export from NetBox
3. Import to Nautobot
4. Check Interface path trace/status and connected_endpoint of interface

Import of NetBox 2.10.5 JSON data

Environment

• Nautobot version: 1.0.0b1
• nautobot-netbox-importer version: 1.0.0

Proposed Functionality

Allow import of NetBox 2.10.5 JSON data.

Use Case

Migration from NetBox 2.10.5.

Sorry for posting this as a feature request but I have no idea where to ask this question. We're currently on NetBox 2.10.5 but would like to Migrate to Nautobot. Is it possible to migrate from 2.10.5 with the current importer? Thanks!

resync doesn't handle custom field data safely

Environment

• Python version:
• Nautobot version: 1.0.0b3
• nautobot-netbox-importer version: 1.1.0

Expected Behavior

When a NetBox dump contains custom field data, re-running the import process after an initial import should succeed with minimal (ideally none) reported changes.

Observed Behavior

Similar to #13 and #14, the re-import may complain about Nautobot failing data validation for records containing custom fields that do not match the latest custom field definitions.

Steps to Reproduce

Various errors when trying to import Netbox data from 3.2.2

Environment

• Python version:
• Nautobot version:
• nautobot-netbox-importer version:

Expected Behavior

Import data fine

Observed Behavior

Redis errors

Steps to Reproduce

1. Export data from current netbox 3.2.2:

python netbox/manage.py dumpdata
--traceback --format=json
--exclude admin.logentry --exclude sessions.session
--exclude extras.ObjectChange --exclude extras.Script --exclude extras.Report
> /tmp/netbox_data.json
2. Setup nautobot on fresh server as per documentation (Install pre-requests + nautobox upto the nautobot-server migrate step
3. copy the netbox data to new server
4. nautobot-server import_netbox_json /tmp/netbox_data.json 2.10.3
5. Random redis error - full error attached

log.txt

VirtualMachine Model Field VCPU is Decimal in Netbox, but int in Nautobot

Environment

• Python version: 3.10
• Nautobot version: 2.10.8
• nautobot-netbox-importer version: 1.4.2

Expected Behavior

Virtual Machines get imported

Observed Behavior

17:36:30 error    Invalid data according to internal data model
  comment: This may be an issue with your source data or may reflect a bug in this plugin.
  action: load
  exception: 1 validation error for VirtualMachine
vcpus
  value is not a valid integer (type=type_error.integer)
  model: virtualmachine
  model_data:
    {'_name': 'testvm',
   $ REDACTED $
     'vcpus': '1.00'}

Reasoning

Netbox Model: https://github.com/netbox-community/netbox/blob/develop/netbox/virtualization/models/virtualmachines.py#L102
Nautobot Model: https://github.com/nautobot/nautobot/blob/develop/nautobot/virtualization/models.py#L304

Proposed solution

This needs some type casting in the migration code, I assume?

Crash with RelatedObjectDoesNotExist: DeviceBay has no device

Environment

• Python version: 3.8.5
• Nautobot version: 1.0.0b2
• nautobot-netbox-importer version: 1.0.1

Expected Behavior

Successful import!

Observed Behavior

Traceback (most recent call last):
  File "/home/jathan/.local/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/home/jathan/.local/lib/python3.8/site-packages/nautobot/core/cli.py", line 56, in main
    run_app(
  File "/home/jathan/.local/lib/python3.8/site-packages/nautobot/core/runner/runner.py", line 268, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/home/jathan/.local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/home/jathan/.local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/jathan/.local/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/jathan/.local/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/home/jathan/.local/lib/python3.8/site-packages/nautobot_netbox_importer/management/commands/import_netbox_json.py", line 92, in handle
    target.sync_from(source, flags=DiffSyncFlags.SKIP_UNMATCHED_DST)
  File "/home/jathan/.local/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 314, in sync_from
    return super().sync_from(source, diff_class=diff_class, flags=flags)
  File "/home/jathan/.local/lib/python3.8/site-packages/diffsync/__init__.py", line 464, in sync_from
    result = syncer.perform_sync()
  File "/home/jathan/.local/lib/python3.8/site-packages/diffsync/helpers.py", line 251, in perform_sync
    changed |= self.sync_diff_element(element)
  File "/home/jathan/.local/lib/python3.8/site-packages/diffsync/helpers.py", line 283, in sync_diff_element
    changed, modified_model = self.sync_model(model, ids, attrs)
  File "/home/jathan/.local/lib/python3.8/site-packages/diffsync/helpers.py", line 329, in sync_model
    model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs)
  File "/home/jathan/.local/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 227, in create
    record = cls.create_nautobot_record(cls.nautobot_model(), nautobot_ids, nautobot_attrs, multivalue_attrs)
  File "/home/jathan/.local/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 190, in create_nautobot_record
    record.clean()
  File "/home/jathan/.local/lib/python3.8/site-packages/nautobot/extras/plugins/validators.py", line 20, in wrapper
    model_clean_func(model_instance)
  File "/home/jathan/.local/lib/python3.8/site-packages/nautobot/dcim/models/device_components.py", line 807, in clean
    if not self.device.device_type.is_parent_device:
  File "/home/jathan/.local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 197, in __get__
    raise self.RelatedObjectDoesNotExist(
nautobot.dcim.models.device_components.RelatedObjectDoesNotExist: DeviceBay has no device.

Steps to Reproduce

1. nautobot-server import_netbox_json /tmp/netbox_data.json 2.10.3

This import is relatively large @ ~280MB. It's a beast.

Duplicate vlangroup import

Environment

• Python version: 3.8.5
• Nautobot version: 1.0.0b2
• nautobot-netbox-importer version: 1.0.1
• Container: nautobot-lab

Expected Behavior

Each vlan-group imported once.

"model": "ipam.vlangroup",
 "pk": 1,

Observed Behavior

Plugin attempted to import the first vlan-group in the dataset multiple times.

12:54:58 Loading all vlangroup records...
12:54:58 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=vlangroup "None__Emeryville" model_id=None__Emeryville
12:54:58 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=vlangroup "None__Emeryville" model_id=None__Emeryville
12:54:58 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=vlangroup "None__Emeryville" model_id=None__Emeryville

Steps to Reproduce

1. nautobot-server import_netbox_json data.json 2.10.3

Pydantic validation errors reported on import of Service and PowerOutletTemplate models

Environment

• Python version: n/a
• Nautobot version: 1.0.0b1
• nautobot-netbox-importer version: 1.0.0

(Spun off from a comment on #5)

Expected Behavior

Successful import of data with no errors or warning messages.

Observed Behavior

I had this similar error pop up many time with services and PowerOutletTemplates.

services example log message:

21:25:37 Invalid data according to internal data model. This may be an issue with your source data or may reflect a bug in this plugin. action=load model=<class 'nautobot_netbox_importer.diffsync.models.ipam.Service'> model_data={'created': '2020-10-09', 'last_updated': '2020-10-09T12:48:45.670Z', 'custom_field_data': {}, 'device': {'site': {'name': 'Equinix WA1'}, 'tenant': {'name': 'MegaFon'}, 'name': 'Megafon_WA1_Perle', 'rack': {'group': None, 'name': '0206'}, 'position': None, 'face': '', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Perle'}, 'model': 'IOLAN SDS2'}, 'device_role': {'name': 'Console Server'}, 'platform': {'name': 'Linux'}, 'serial': '', 'asset_tag': None, 'cluster': None}, 'virtual_machine': None, 'name': 'Serial1', 'protocol': 'tcp', 'ports': '["10001"]', 'description': 'Console for re0', 'ipaddresses': [{'address': IPNetwork('123.456.789.123/32'), 'vrf': None, 'tenant': None, 'assigned_object_type': {'app_label': 'dcim', 'model': 'interface'}, 'assigned_object_id': {'device': {'site': {'name': 'Equinix WA1'}, 'tenant': {'name': 'MegaFon'}, 'name': 'Megafon_WA1_Perle', 'rack': {'group': None, 'name': '0206'}, 'position': None, 'face': '', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Perle'}, 'model': 'IOLAN SDS2'}, 'device_role': {'name': 'Console Server'}, 'platform': {'name': 'Linux'}, 'serial': '', 'asset_tag': None, 'cluster': None}, 'name': 'eth0'}}], 'pk': 4}
1 validation error for Service
ports
  value is not a valid list (type=type_error.list)

Not sure which value is supposed to be a valid list here..

PowerOutletTemplate example log message:

21:25:17 Invalid data according to internal data model. This may be an issue with your source data or may reflect a bug in this plugin. action=load model=<class 'nautobot_netbox_importer.diffsync.models.dcim.PowerOutletTemplate'> model_data={'device_type': {'manufacturer': {'name': 'Raritan'}, 'model': 'PX3-5497V'}, 'name': 'Outlet24', '_name': 'Outlet00000024', 'label': '', 'description': '', 'type': 'iec-60320-c19', 'power_port': {'device_type': {'manufacturer': {'name': 'Raritan'}, 'model': 'PX3-5497V'}, 'name': 'Inlet1'}, 'feed_leg': 'C', 'pk': 804}
1 validation error for PowerOutletTemplate
type
  instance of staticmethod expected (type=type_error.arbitrary_type; expected_arbitrary_type=staticmethod)

In this case I'm also not sure which value its complaining about unfortunately.

data import produces failures for existing users (e.g. admin at least)

Environment

• Python version: 3.8.10
• Nautobot version: 1.0.3
• nautobot-netbox-importer version: 1.4.0

Expected Behavior

no failures when importing data

Observed Behavior

existing users cause failures, even if no users are there beyond admin, the existence of admin causes a failure

Steps to Reproduce

1. Create a working installation of Nautobot 1.0.3 on Ubuntu 20.04 with nautobot-netbox-importer 1.4.0 following installation guide for Nautobot on Ubuntu and installation guide from plugin github page.
2. Export data from a working installation of Netbox 2.10.4 following instructions on the plugin github page
3. Import data with the use of nautobot-netbox-importer following instructions on the plugin github page

Duplicate address warning on VRRP addresses

Environment

• Python version: 3.6.8
• Nautobot version: 1.0.0b2
• nautobot-netbox-importer version: 1.0.1

Expected Behavior

Import data from NetBox 2.10.7

Observed Behavior

14:37:39 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=ipaddress "10.99.21.129/26__{'name': 'MGMT', 'rd': '64512:', 'tenant': {'name': 'Tenant'}}__{'name': 'Tenant'}__None__None" model_id=10.99.21.129/26__{'name': 'MGMT', 'rd': '64512:', 'tenant': {'name': 'Tenant'}}__{'name': 'Tenant'}__None__None

Note the above address is indeed duplicated, as it is a VRRP address, and VRRP addresses are permitted to be duplicated in NetBox

Steps to Reproduce

1. $ sudo -u postgres psql
2. postgres=# DROP DATABASE nautobot;
3. postgres=# CREATE DATABASE nautobot;
4. postgres=# GRANT ALL PRIVILEGES ON DATABASE nautobot TO nautobot;
5. $ sudo systemctl restart nautobot nautobot-worker
6. $ nautobot-server migrate
7. $ nautobot-server createsuperuser
8. $ nautobot-server collectstatic
9. $ nautobot-server check
10. $ python netbox/manage.py dumpdata --traceback --format=json --exclude admin.logentry --exclude sessions.session --exclude extras.ObjectChange --exclude extras.Script --exclude extras.Report > /tmp/netbox_data.json
11. $ nautobot-server import_netbox_json /tmp/netbox_data.json 2.10.5

Development Setup - invoke does not start the docs service.

Environment

• Python version: 3.9 (but 3.7 inside the docker container)
• Nautobot version: 1.5.10
• nautobot-netbox-importer version: Dev environment

Expected Behavior

https://docs.nautobot.com/projects/netbox-importer/en/latest/dev/dev_environment/ suggests that invoke start should start the mkdocs container as well as the Nautobot container.

Observed Behavior

It doesn't start

Steps to Reproduce

1. Follow the steps in the document.
2. Run invoke start
3. Try to access http://localhost:8001/

The container is not running:

CONTAINER ID   IMAGE                                           COMMAND                  CREATED          STATUS                      PORTS                    NAMES
350c4f8271ca   nautobot-netbox-importer/nautobot:1.3.7-py3.7   "nautobot-server rqw…"   16 minutes ago   Up 16 minutes               8080/tcp                 nautobot-netbox-importer_worker_1
c3910a9d9602   nautobot-netbox-importer/nautobot:1.3.7-py3.7   "/docker-entrypoint.…"   16 minutes ago   Up 16 minutes (unhealthy)   0.0.0.0:8080->8080/tcp   nautobot-netbox-importer_nautobot_1
e62569e2faab   postgres:13-alpine                              "docker-entrypoint.s…"   16 minutes ago   Up 16 minutes               0.0.0.0:5432->5432/tcp   nautobot-netbox-importer_postgres_1
16f74cb8e1e0   redis:6-alpine                                  "docker-entrypoint.s…"   16 minutes ago   Up 16 minutes               0.0.0.0:6379->6379/tcp   nautobot-netbox-importer_redis_1
0f75e158520e   rediscommander/redis-commander:latest           "/usr/bin/dumb-init …"   13 days ago      Up 13 days (healthy)        0.0.0.0:8081->8081/tcp   redis-local-redis-commander-1

Missing custom field error still may be seen on second-pass sync

Environment

• Python version:
• Nautobot version: 1.0.0b3
• nautobot-netbox-importer version: 1.2.0

Expected Behavior

Now that #35 is fixed, we should be able to successfully import records with and without custom fields.

Observed Behavior

If a Device or IPAddress is missing a required CustomField, then on the second-pass of the importer (when trying to set primary_ip4 on a Device or nat_inside on an IPAddress), Nautobot may still report data validation errors due to missing required custom fields:

17:33:56 error    Nautobot reported a data validation error - check your source data
  action: update
  exception: ["Missing required custom field 'my_required_field'."]
  model: <class 'nautobot.dcim.models.devices.Device'>
  model_data: 
    {'pk': UUID('d055b246-9713-597b-9bd1-2e4002a4ffd8'),
     'primary_ip4': <IPAddress: 10.1.1.1/24>}

17:34:03 error    Nautobot reported a data validation error - check your source data
  action: update
  exception: ["Missing required custom field 'another_required_field'."]
  model: <class 'nautobot.ipam.models.IPAddress'>
  model_data: 
    {'nat_inside': <IPAddress: 10.1.1.1/28>,
     'pk': UUID('5a9a92c5-ee2b-5deb-aa73-c03d67603ed3')}

Steps to Reproduce

1. Clear the Nautobot database
2. Ensure that the NetBox database dump defines required CustomFields on Device and/or IPAddress
3. Ensure that the NetBox database dump contains:
   a. Device with an assigned primary_ip4 address but no value for the required custom field(s), and/or
   b. IPAddress with an assigned nat_inside address but no value for the required custom field(s)
4. Attempt to import the NetBox database dump into Nautobot using this plugin.

Pylint test fails, due poetry lock dependency issue

Environment

• Python version:
• Nautobot version:
• nautobot-netbox-importer version: develop branch

Expected Behavior

Pylint test succeeds

Observed Behavior

Run poetry lock
Updating dependencies
Resolving dependencies...
  SolverProblemError
  The current project's Python requirement (>=3.6,<4.0) is not compatible with some of the required packages Python requirement:
    - black requires Python >=3.6.2, so it will not be satisfied for Python >=3.6,<3.6.2
    - black requires Python >=3.6.2, so it will not be satisfied for Python >=3.6,<3.6.2
    - black requires Python >=3.6.2, so it will not be satisfied for Python >=3.6,<3.6.2
  Because no versions of black match <22.1.0 || >22.1.0,<22.3.0 || >22.3.0,<22.6.0 || >22.6.0
   and black (22.1.0) requires Python >=3.6.2, black is forbidden.
  And because black (22.3.0) requires Python >=3.6.2, black is forbidden.
  So, because black (22.6.0) requires Python >=3.6.2
   and nautobot-netbox-importer depends on black (*), version solving failed.
  at /opt/hostedtoolcache/Python/3.9.[13](https://github.com/nautobot/nautobot-plugin-netbox-importer/runs/7108475588?check_suite_focus=true#step:5:14)/x64/lib/python3.9/site-packages/poetry/puzzle/solver.py:241 in _solve
      [23](https://github.com/nautobot/nautobot-plugin-netbox-importer/runs/7108475588?check_suite_focus=true#step:5:24)7│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      [24](https://github.com/nautobot/nautobot-plugin-netbox-importer/runs/7108475588?check_suite_focus=true#step:5:25)0│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes
  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties
    For black, a possible solution would be to set the `python` property to ">=3.6.2,<4.0"
    For black, a possible solution would be to set the `python` property to ">=3.6.2,<4.0"
    For black, a possible solution would be to set the `python` property to ">=3.6.2,<4.0"
   [ https://python-poetry.org/docs/dependency-specification/#python-restricted-dependen](https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies)ci
   [ https://python-poetry.org/docs/dependency-specification/#using-environment-mar](https://python-poetry.org/docs/dependency-specification/#using-environment-markers)
Error: Process completed with exit code 1.

Steps to Reproduce

Add support for importing NetBox 3.2.x data

With #91 as a pre-requisite, the following NetBox 3.2.x changes need to be accounted for by the importer:

• Additional enum values (netbox-community/netbox#9110, netbox-community/netbox#9193, netbox-community/netbox@9e9e90f, netbox-community/netbox#9982, etc.)
• Added Module and ModuleTypemodels and related foreign-keys - no current equivalent in Nautobot
• Added InventoryItemRole model and InventoryItem.role field - no current equivalent in Nautobot, though generic Roles in Nautobot 2.0 may fill some of this space
• Added InventoryItemTemplate model - no current equivalent in Nautobot
• Added ServiceTemplate model - no current equivalent in Nautobot
• Removal of Site.asn, Site.contact_name, Site.contact_phone, and Site.contact_email fields
• CustomField/CustomLink/Tag association to JournalEntry model - Nautobot Note model does not currently support these
• Added ProviderNetwork.service_id field - no current equivalent in Nautobot
• Changed created field on all models from DateField to DateTimeField - coming in Nautobot 2.0, will need to be type-converted for now.
• Added InventoryItem.component field - no Nautobot equivalent
• Added ConfigContext.cluster_types field - no Nautobot equivalent
• Added Interface.vrf field - no Nautobot equivalent
• Added Interface.speed and Interface.duplex fields - no Nautobot equivalents
• Added VLANGroup.min_vid and VLANGroup.max_vid fields - no Nautobot equivalents
• Added CustomLink.enabled field - no Nautobot equivalent
• Added object and multi-object custom fields - these should be translatable to Nautobot Relationship and RelationshipAssociation records.

Command errors out with ObjectNotFound exception

Environment

• Python version: 3.7
• Nautobot version: 1.0.0b1
• nautobot-netbox-importer version: 1.0.0

Expected Behavior

Import should run to completion successfully.

Observed Behavior

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 5, in <module>
    main()
  File "/opt/nautobot/nautobot/core/cli.py", line 62, in main
    initializer=_configure_settings,  # Called after defaults
  File "/opt/nautobot/nautobot/core/runner/runner.py", line 268, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.7/site-packages/nautobot_netbox_importer/management/commands/import_netbox_json.py", line 91, in handle
    target.sync_from(source, flags=DiffSyncFlags.SKIP_UNMATCHED_DST)
  File "/usr/local/lib/python3.7/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 299, in sync_from
    return super().sync_from(source, diff_class=diff_class, flags=flags)
  File "/usr/local/lib/python3.7/site-packages/diffsync/__init__.py", line 464, in sync_from
    result = syncer.perform_sync()
  File "/usr/local/lib/python3.7/site-packages/diffsync/helpers.py", line 251, in perform_sync
    changed |= self.sync_diff_element(element)
  File "/usr/local/lib/python3.7/site-packages/diffsync/helpers.py", line 283, in sync_diff_element
    changed, modified_model = self.sync_model(model, ids, attrs)
  File "/usr/local/lib/python3.7/site-packages/diffsync/helpers.py", line 329, in sync_model
    model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs)
  File "/usr/local/lib/python3.7/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 192, in create
    diffsync_ids, nautobot_ids = cls.clean_ids(diffsync, ids)
  File "/usr/local/lib/python3.7/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 168, in clean_ids
    return cls.clean_ids_or_attrs(diffsync, ids)
  File "/usr/local/lib/python3.7/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 114, in clean_ids_or_attrs
    target_record = diffsync.get(target_class, diffsync_value)
  File "/usr/local/lib/python3.7/site-packages/diffsync/__init__.py", line 567, in get
    raise ObjectNotFound(f"{modelname} {uid} not present in {self.name}")
diffsync.exceptions.ObjectNotFound: interface {'site': {'name': 'some site'}, 'tenant': None, 'name': 'some device'}__mgmt0 not present in          NautobotDiffSync

Steps to Reproduce

Details of the data being imported are unavailable at this time.

No data validation for legacy custom fields on an object

Environment

• Python version: 3.8.5
• Nautobot version: 1.0.0b2
• nautobot-netbox-importer version: 1.0.1
• Container: nautobot-lab

Expected Behavior

Objects with redundant/legacy custom fields in the source data, that are no longer applied to the model should be imported. Perhaps a log could be printed to highlight the field(s) ignored during import.

Observed Behavior

Data validation error.

13:03:30 Nautobot reported a data validation error - check your source data action=create model=<class 'nautobot.ipam.models.IPAddress'>
..
["Unknown field name 'customfield_22' in custom field data."]

Steps to Reproduce

1. nautobot-server import_netbox_json data.json 2.10.3

Add support for importing NetBox 3.1.x data

With #90 as a pre-requisite, the following NetBox 3.1.x changes need to be accounted for by the importer:

• Port type and other enum additions/changes (netbox-community/netbox@b7b5a57, netbox-community/netbox#8269, netbox-community/netbox@2b6e040, netbox-community/netbox@df43039, etc.)
• ObjectChange records for ImageAttachment objects
  • Not currently valid in Nautobot
• Many changes and additions to Interface fields
  • Added bridge field - equivalent functionality exists in Nautobot 1.4 and later and support should be addedW
  • Added rf_* fields, tx_power field, wwn field - no Nautobot equivalents at present
  • Renamed cable_peer and cable_peer_type fields (unchanged in Nautobot)
• JSON CustomField type - supported in Nautobot 1.3 and later
• Loosened uniqueness constraints of Cluster, Region, SiteGroup, and RackGroup Location models (non-globally-unique name in particular)
  • SiteGroup has no equivalent in Nautobot at this time and can be ignored for now
  • Cluster, Region and RackGroup may need to account for currently stricter uniqueness criteria in Nautobot
• RackGroup Location.tenant and Cable.tenant fields - no current equivalent in Nautobot
• longtext CustomField type - no current equivalent in Nautobot
• Tags for "organizational" models - slated for Nautobot 2.0, but will need to be discarded at present
• Device.airflow and DeviceType.airflow fields - no current equivalent in Nautobot
• Added ASN model - no current equivalent in Nautobot core
• Webhook.condition field - no current equivalent in Nautobot
• Added FHRPGroup and FHRPGroupAssignment models - no current equivalent in Nautobot
• Added ConfigRevision model - no current equivalent in Nautobot
• Added WirelessLAN, WirelessLANGroup, and WirelessLink models - no current equivalent in Nautobot
• Added Contact, ContactGroup, ContactRole, and ContactAssignment models - no current equivalent in Nautobot
• Support for negative validation_minimum and validation_maximum values on CustomField - should "just work" but needs testing
• Loosened IPAddress.dns_name regex validation, may need to account for names that are not currently permitted in Nautobot

Instructions for using this plugin with the docker environments

Environment

• Nautobot version: n/a
• nautobot-netbox-importer version: n/a

Proposed Functionality

Instructions for use with netbox/nautobot in docker environments

Use Case

netbox/nautobot in docker

Add option to skip clean() validation on imported data

Environment

• Nautobot version: v1.0.2
• nautobot-netbox-importer version: 1.30

Proposed Functionality

Would be great to add an option to the importer to skip running Nautobot’s clean() validation on the imported data, which would mean that we could successfully import erroneous data from NetBox and have it be just as (in)valid as it was there.

Use Case

I have Netbox --> Nautobot DB dependency where we cannot have access to Netbox so we need to accept all the data as they are in Netbox

EUI-type MAC addresses cannot be imported

Environment

• Python version: 3.7.3
• Nautobot version: 1.0.0b2
• nautobot-netbox-importer version: develop branch @ April 1st

Expected Behavior

The MAC is imported for a given interface.

Observed Behavior

The MAC address associated with the interface failed to import, we observed this on data associated with our "older" devices.

08:37:07 Invalid data according to internal data model. This may be an issue with your source data or may reflect a bug in this plugin. action=load model=interface model_data={'id': UUID('fcead997-0ef8-aaaa-aaaa-21cbdb588e34'), 'custom_field_data': {}, 'device': {'site': {'name': 'AAAA'}, 'tenant': {'name': 'AS12345'}, 'name': 'some.old.device.net', 'rack': {'group': {'site': {'name': 'AAAA'}, 'name': 'SiteName'}, 'name': 'GroupName'}, 'position': 40, 'face': 'rear', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Sysco Systems'}, 'model': 'Sysco 1111'}, 'device_role': {'name': 'Old Crap'}, 'platform': None, 'serial': 'FTXAAAAAA', 'asset_tag': None, 'cluster': None}, 'name': 'ge0/0', 'label': '', 'description': '', '_cable_peer_type': None, '_cable_peer_id': None, '_path': None, 'enabled': True, 'mac_address': EUI('00:12:34:56:78:9A'), 'mtu': 1500, 'mode': '', '_name': '0000999999999999ge000000............', 'type': '1000base-t', 'mgmt_only': False, 'tags': [], 'ip_addresses': <django.contrib.contenttypes.fields.create_generic_related_manager.<locals>.GenericRelatedObjectManager object at 0x7fb2e4050f60>, '_cabled_as_a': <django.contrib.contenttypes.fields.create_generic_related_manager.<locals>.GenericRelatedObjectManager object at 0x7fb2e2447358>, '_cabled_as_b': <django.contrib.contenttypes.fields.create_generic_related_manager.<locals>.GenericRelatedObjectManager object at 0x7fb2e24479e8>, 'tagged_items': <django.contrib.contenttypes.fields.create_generic_related_manager.<locals>.GenericRelatedObjectManager object at 0x7fb2e2447780>, 'pk': UUID('fcead997-0ef8-aaaa-aaaa-21cbdb588e34')}
1 validation error for Interface
mac_address
  str type expected (type=type_error.str)

Steps to Reproduce

Not known / might be legacy NetBox data structures involved.

Our production NetBox instance has data in it that's over a year old; data exported from v2.10.5.

importer logs an `error` when trying to create a `custom_link` to a Secret model which is removed in Nautobot

Environment

• Python version: 3.9.1
• Nautobot version: 1.0.2
• nautobot-netbox-importer version: 1.0.3

Expected Behavior

Imports to Secret should be skipped

Observed Behavior

importer return error when trying to create a custom_link to a Secret model which is removed in Nautobot

10:42:06 error    Nautobot reported a database integrity error
  action: create
  exception: null value in column "content_type_id" violates not-null constraint
DETAIL:  Failing row contains (f8f14a85-390f-59d1-937c-e8e0e94ddc47, 2021-06-23, 2021-06-23 10:42:06.834875+00, HP NA, HP-NA, {% if obj.primary_ip %}https://im-na.wdf.sap.corp/search.do?devi..., 100, CND, default, t, null).
  model: <class 'nautobot.extras.models.models.CustomLink'>
  model_data:
    {'button_class': 'default',
     'content_type': None,
     'created': None,
     'group_name': 'CND',
     'name': 'HP NA',
     'new_window': True,
     'pk': UUID('f8f14a85-390f-59d1-937c-e8e0e94ddc47'),
     'target_url': '{% if obj.primary_ip '
                   '%}https://im-na.wdf.sap.corp/search.do?device={{ '
                   'obj.primary_ip.address.ip '
                   '}}&pageTitle=Device+Search+Results&fo_device=contains&filterName=searchdevice&fo_managementStatus=equals&managementStatus=0&managementStatus=1&managementStatus=3&mergeMode=replace&basic=true&sortBy=hostName&filterType=Device&quickSearch=1{% '
                   'endif %}',
     'text': 'HP-NA',
     'weight': 100}

ValidationError crash on rack import

Environment

• Python version: 3.8.8
• Nautobot version: 1.0.0b1
• nautobot-netbox-importer version: 1.0.0

Expected Behavior

Full import!

I have a rather large production NetBox 2.10.3 dump (about 600 MB, ~40k devices, ~280k interfaces, ~88k cables, ~90k its, ~14k prefixes) I'm trying to import.

Observed Behavior

Crashed about ~1% of the way through import.

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/cli.py", line 55, in main
    run_app(
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/runner/runner.py", line 268, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/management/commands/import_netbox_json.py", line 91, in handle
    target.sync_from(source, flags=DiffSyncFlags.SKIP_UNMATCHED_DST)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 299, in sync_from
    return super().sync_from(source, diff_class=diff_class, flags=flags)
  File "/usr/local/lib/python3.8/site-packages/diffsync/__init__.py", line 464, in sync_from
    result = syncer.perform_sync()
  File "/usr/local/lib/python3.8/site-packages/diffsync/helpers.py", line 251, in perform_sync
    changed |= self.sync_diff_element(element)
  File "/usr/local/lib/python3.8/site-packages/diffsync/helpers.py", line 283, in sync_diff_element
    changed, modified_model = self.sync_model(model, ids, attrs)
  File "/usr/local/lib/python3.8/site-packages/diffsync/helpers.py", line 329, in sync_model
    model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 203, in create
    record = cls.create_nautobot_record(cls.nautobot_model(), nautobot_ids, nautobot_attrs, multivalue_attrs)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 180, in create_nautobot_record
    record.clean()
  File "/usr/local/lib/python3.8/site-packages/nautobot/extras/plugins/validators.py", line 20, in wrapper
    model_clean_func(model_instance)
  File "/usr/local/lib/python3.8/site-packages/nautobot/dcim/models/devices.py", line 586, in clean
    raise ValidationError(
django.core.exceptions.ValidationError: {'rack': ['Rack ABC03 (CENSORED) does not belong to site iad1.']}

Looking at NetBox from which the dump was exported, that particular rack is in site "ord1", so the error is seemingly accurate but I'm not clear on the context otherwise.

Steps to Reproduce

1. Receive my massive database dump "somehow"
2. nautobot-server import_netbox_json /tmp/netbox_data.json 2.10.3 and then wait
3. Crash!

Make --bypass-data-validation able to select models for which data validation should be disabled

Environment

• Nautobot version: n/a
• nautobot-netbox-importer version: n/a

Proposed Functionality

Enhance the --bypass-data-validation function to be able to select the models for which validation should be disabled

Use Case

There are certain models, who can only be imported when ignoring certain constraints (e.g. the VLAN Group has in nautobot a site, and the site of a VLAN in the Group must match the site, while netbox 2.11 does not have a site attribute for a VLAN group).

I consider it useful to be able to disable the validation only for those problematic models, and maybe change the error on such an event to INFO, so that you easily still see issues with other models, and can filter for them.

Saving logs to file produces escaped characters

Environment

• Python version: 3.6.8
• Nautobot version: 1.0.0b3
• nautobot-netbox-importer version: 1.1.0
• system: CentOS Linux release 7.8.2003 (Core)

Expected Behavior

I want tot have logs saved and readable for future reference.
I merge stderr and stdout to single pipe and tee to log file to be able to see the progress as it takes couple of hours to import the whole DB.

Observed Behavior

Logs contain escaped characters making reading them harder.

It seems to be caused by colorama, but using parameter --no-color is ignored altogether. In console output I can see color correctly, but the same is saved to logfile with the escaped characters.

Steps to Reproduce

1. docker exec -it nautobot nautobot-server import_netbox_json /opt/db_dump.json 2.10.3 2>&1 | tee test.log

Traceback: diffsync.exceptions.ObjectAlreadyExists: Object None__<object>__ already present

Environment

• Python version: 3.6.8
• Nautobot version: 1.0.0b2
• nautobot-netbox-importer version: 1.0.1

Expected Behavior

Import data from NetBox 2.10.7

Observed Behavior

...
14:00:00 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=virtualchassis "None__switch1__" model_id=None__switch1__
...
14:01:50 Data loading from Nautobot complete.
14:01:50 Beginning data synchronization...
14:02:15 Nautobot reported a data validation error - check your source data action=create model=<class 'nautobot.dcim.models.devices.Device'> model_data={'site': <Site: Site>, 'tenant': <Tenant: Tenant>, 'name': 'switch1b', 'rack': <Rack: B2.2 (B2.2)>, 'position': 36, 'face': 'front', 'vc_position': 2, 'vc_priority': None, 'device_type': <DeviceType: N4032>, 'device_role': <DeviceRole: VPS Switch>, 'platform': <Platform: Dell NOS 6>, 'serial': '', 'asset_tag': None, 'cluster': None, 'custom_field_data': {'management_tool': 'Salt'}, 'status': 
<Status: Active>, 'local_context_data': None, 'virtual_chassis': None, 'primary_ip4': None, 'primary_ip6': None, 'comments': ''}
{'position': ['U36 is already occupied or does not have sufficient space to accommodate this device type: N4032 (1U)']}
14:02:15 device create did not return the model object. action=create diffs={'+': {'custom_field_data': {'management_tool': 'Salt'}, 'status': {'slug': 'active'}, 'local_context_data': None, 'virtual_chassis': {'master': {'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'switch1b', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'B2.2'}, 'position': 36, 'face': 'front', 'vc_position': 2, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Dell'}, 'model': 'N4032'}, 'device_role': {'name': 'VPS Switch'}, 'platform': {'name': 'Dell NOS 6'}, 'serial': '', 'asset_tag': None, 'cluster': None}, 'name': 'switch1', 'domain': ''}, 'primary_ip4': None, 'primary_ip6': None, 'comments': ''}} dst=<NautobotDiffSync> flags=<DiffSyncFlags.SKIP_UNMATCHED_DST: 4> model=device src=<NetBox210DiffSync> status=failure unique_id={'name': 'Site'}__{'name': 'Tenant'}__switch1b__{'group': {'site': 
{'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'B2.2'}__36__front__2__None__{'manufacturer': {'name': 'Dell'}, 'model': 'N4032'}__{'name': 'VPS Switch'}__{'name': 'Dell NOS 6'}____None__None
14:02:15 No object resulted from sync, will not process child objects. action=create diffs={'+': {'custom_field_data': {'management_tool': 'Salt'}, 'status': {'slug': 'active'}, 'local_context_data': None, 'virtual_chassis': {'master': {'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'switch1b', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'B2.2'}, 'position': 36, 'face': 'front', 'vc_position': 2, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Dell'}, 'model': 'N4032'}, 'device_role': {'name': 'VPS Switch'}, 'platform': {'name': 'Dell NOS 6'}, 'serial': '', 'asset_tag': None, 'cluster': None}, 'name': 
'switch1', 'domain': ''}, 'primary_ip4': None, 'primary_ip6': None, 'comments': ''}} dst=<NautobotDiffSync> flags=<DiffSyncFlags.SKIP_UNMATCHED_DST: 4> model=device src=<NetBox210DiffSync> unique_id={'name': 'Site'}__{'name': 'Tenant'}__switch1b__{'group': {'site': 
{'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'B2.2'}__36__front__2__None__{'manufacturer': {'name': 'Dell'}, 'model': 'N4032'}__{'name': 'VPS Switch'}__{'name': 'Dell NOS 6'}____None__None
Traceback (most recent call last):
  File "/opt/nautobot/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot/core/cli.py", line 63, in main
    initializer=_configure_settings,  # Called after defaults
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot/core/runner/runner.py", line 268, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/opt/nautobot/lib64/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/opt/nautobot/lib64/python3.6/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/nautobot/lib64/python3.6/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/nautobot/lib64/python3.6/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot_netbox_importer/management/commands/import_netbox_json.py", line 92, in handle 
    target.sync_from(source, flags=DiffSyncFlags.SKIP_UNMATCHED_DST)
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 314, in sync_from
    return super().sync_from(source, diff_class=diff_class, flags=flags)
  File "/opt/nautobot/lib64/python3.6/site-packages/diffsync/__init__.py", line 464, in sync_from
    result = syncer.perform_sync()
  File "/opt/nautobot/lib64/python3.6/site-packages/diffsync/helpers.py", line 251, in perform_sync
    changed |= self.sync_diff_element(element)
  File "/opt/nautobot/lib64/python3.6/site-packages/diffsync/helpers.py", line 293, in sync_diff_element
    self.dst_diffsync.add(model)
  File "/opt/nautobot/lib64/python3.6/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 230, in add
    super().add(obj)
  File "/opt/nautobot/lib64/python3.6/site-packages/diffsync/__init__.py", line 623, in add
    raise ObjectAlreadyExists(f"Object {uid} already present")
diffsync.exceptions.ObjectAlreadyExists: Object None__switch1__ already present

Steps to Reproduce

1. $ python netbox/manage.py dumpdata --traceback --format=json --exclude admin.logentry --exclude sessions.session --exclude extras.ObjectChange --exclude extras.Script --exclude extras.Report > /tmp/netbox_data.json
2. $ nautobot-server import_netbox_json /tmp/netbox_data.json 2.10.5

Allow "duplicate" objects for NULL!=NULL case and others

Environment

• Nautobot version: any
• nautobot-netbox-importer version: 1.0.1

Background

unique_together and NULL != NULL

In SQL, NULL != NULL, which means that any unique_together constraint in Django will not apply as one might intuitively expect when any one or more of the fields in the constraint can potentially be NULL. For example, InventoryItem has a unique_together of ["device", "parent", "name"], but parent can be NULL, meaning that it is perfectly permissible from a database perspective to have multiple InventoryItem records with the same device and name so long as parent is NULL. (By contrast, intuitively, the database will not allow the same device, name, and non-NULL parent on multiple InventoryItem records.)

Because, in Python, we represent NULL as None, and in Python None == None, the DiffSync uniqueness check between multiple records with NULL values for any of their identifiers will fail, and DiffSync will consider these records to be invalid duplicate/redundant records and will discard/disregard the "extras".

The workaround that we've used thus far (#6, #19, etc.) has been to move more of the model fields into the DiffSync identifiers list, such as serial number and asset tag, in hopes of disambiguating between multiple such records, but this is a workaround only.

Non-unique records in general

Some NetBox/Nautobot data models (such as IPAddress) have no defined unique_together constraints and may be duplicated freely -- in some cases this absolutely a desired feature, such as in the case of VRRP addresses (#27). Even disregarding the NULL != NULL behavior described previously, this is another case where the importer plugin's (and DiffSync's) handling of duplicate records needs to be improved/fixed.

Proposed Functionality

Implement a way to store multiple NetBox/Nautobot records under a single DiffSync "unique id", handling both the coalescing of multiple NetBox JSON records into a single DiffSync record as well as the fan-out of a single DiffSync record into multiple Nautobot database records.

This design will need to be able to handle references to these specific objects as well (for example, assignment of duplicated VRRP addresses to different device interfaces).

Use Case

This would handle both:

1. Multiple records that coexist despite defined unique_together conditions (#11, #19, #26, etc.) -- handling NULL!=NULL as described above
2. Multiple records that coexist in the absence of any defined unique_together conditions (such as IPAddress #27).

Object's creation date inconsistency after db-migration

Environment

nautobot@mia:~$ pip freeze | grep nautobot
nautobot==1.0.1
nautobot-netbox-importer==1.3.0

Expected Behavior

Object's changelog shows consistent created date for imported objects.
From the user perspective, I would like to see following fields :

• a new entry in the object's Changelog: DB-Migrated indicating object's migration date
• migrated object inheriting the creation date

Observed Behavior

Object's changelog shows inconsistent created date for imported objects. In the picture below, two different created dates are displayed to the users, which might be confusing:

Steps to Reproduce

1. Import NetBox data with importer including the ChangeLog.

Change log does not keep original times and users

Environment

• Python version: 3.8.10
• Nautobot version: 1.0.3
• nautobot-netbox-importer version: 1.4.0

Expected Behavior

Change log to appear with original times and users

Observed Behavior

In change log change times are modified as equal to the import time, all are the same.
In change log users are appearing as undefined.

Steps to Reproduce

1. Create a working installation of Nautobot 1.0.3 on Ubuntu 20.04 with nautobot-netbox-importer 1.4.0 following installation guide for Nautobot on Ubuntu and installation guide from plugin github page.
2. Export data from a working installation of Netbox 2.10.4 following instructions on the plugin github page
3. Import data with the use of nautobot-netbox-importer following instructions on the plugin github page

resync falsely detects changes in Status fields

Environment

• Python version:
• Nautobot version: 1.0.0b3
• nautobot-netbox-importer version: 1.1.0

Expected Behavior

After successfully importing a NetBox data dump, a second re-run of the import process with the same data should report no changes detected.

Observed Behavior

Objects with "status" fields are seen as having changes, e.g.:

  src: NetBox210DiffSync
  dst: NautobotDiffSync
  flags: DiffSyncFlags.SKIP_UNMATCHED_DST
  action: update
  model: site
  unique_id: d25b6c47-2eb1-570e-aee0-8de6b100273d
  diffs: 
    {'+': {'status': {'slug': 'active'}},
     '-': {'status': TaggedForeignKeyField('d29cd273-9e32-4b20-8d65-8f39abac73ad')}}

Steps to Reproduce

1. nautobot-server import_netbox_json dumps/netbox_dump.json 2.10.5
2. nautobot-server import_netbox_json dumps/netbox_dump.json 2.10.5 - again

Import of objectpermissions that have constraints consisting of a list of ORed conditions fails

Environment

• Python version: 3.9
• Nautobot version: 1.2.2
• nautobot-netbox-importer version: 1.4.0

Expected Behavior

Import of objectpermissions that have constraints consisting of a list of conditions such as
[{"name": "foo"}, {"name": "bar"}]
should work.

Observed Behavior

objectpermission : 0%| | 0/ 323 [00:00]17:15:54 error Invalid data according to internal data model
12:15:54§ comment: This may be an issue with your source data or may reflect a bug in this plugin.
12:15:54§ action: load
12:15:54§ exception: 1 validation error for ObjectPermission
12:15:54§ constraints
12:15:54§ value is not a valid dict (type=type_error.dict)
12:15:54§ model: objectpermission
12:15:54§ model_data:
12:15:54§ {'actions': '["view"]',
12:15:54§ 'constraints': [{'name': 'foo'}, {'name': 'bar'}],
12:15:54§ 'description': '',
12:15:54§ 'enabled': True,
12:15:54§ 'groups': [{'name': 'foobargrp'}],
12:15:54§ 'name': 'foobarperm',
12:15:54§ 'object_types': [{'app_label': 'tenancy', 'model': 'tenantgroup'},
12:15:54§ {'app_label': 'tenancy', 'model': 'tenant'}],
12:15:54§ 'pk': 308,
12:15:54§ 'users': [UUID('036b093d-d350-538e-bad9-1fd132b1f5fa')]}

The model validation is overly restrictive in allowing only dict type while lists should also be allowed:
https://github.com/nautobot/nautobot-plugin-netbox-importer/blob/a225bc38b9ef51ff8d6f1fdc72dd5472eedf0b30/nautobot_netbox_importer/diffsync/models/users.py#L27

Steps to Reproduce

1. In netbox, create a permission with the following properties
   object_types: tenancy/tenant, tenancy/tenant group
   constraints: [{"name": "foo"}, {"name": "bar"}]

2. Export netbox data

3. Import to nautobot...

pydantic.error_wrappers.ValidationError crash on device import

Environment

• Python version: 3.8.8
• Nautobot version: 1.0.0b1
• nautobot-netbox-importer version: 1.0.0

Expected Behavior

Successful import from the same massive NetBox 2.10.3 dump. Related to #4.

Observed Behavior

Made it further, but crashed on a Device validation?

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/cli.py", line 55, in main
    run_app(
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/runner/runner.py", line 268, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/management/commands/import_netbox_json.py", line 91, in handle
    target.sync_from(source, flags=DiffSyncFlags.SKIP_UNMATCHED_DST)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 299, in sync_from
    return super().sync_from(source, diff_class=diff_class, flags=flags)
  File "/usr/local/lib/python3.8/site-packages/diffsync/__init__.py", line 464, in sync_from
    result = syncer.perform_sync()
  File "/usr/local/lib/python3.8/site-packages/diffsync/helpers.py", line 251, in perform_sync
    changed |= self.sync_diff_element(element)
  File "/usr/local/lib/python3.8/site-packages/diffsync/helpers.py", line 283, in sync_diff_element
    changed, modified_model = self.sync_model(model, ids, attrs)
  File "/usr/local/lib/python3.8/site-packages/diffsync/helpers.py", line 329, in sync_model
    model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 208, in create
    return super().create(diffsync, diffsync_ids, diffsync_attrs)
  File "/usr/local/lib/python3.8/site-packages/diffsync/__init__.py", line 197, in create
    model = cls(**ids, diffsync=diffsync, **attrs)
  File "pydantic/main.py", line 400, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Device
status
  none is not an allowed value (type=type_error.none.not_allowed)

Steps to Reproduce

1.Receive my massive database dump "somehow"
2. nautobot-server import_netbox_json /tmp/netbox_data.json 2.10.3 and then wait
3. Crash!

Pretty-print structlogs

Environment

• Nautobot version:
• nautobot-netbox-importer version: 1.0.1

Proposed Functionality

Add pretty-printing to the plugin's log output, something like:

11:53:16 Nautobot reported an error about a missing required object
  action=update
  model=<class 'nautobot.dcim.models.devices.Device'>
  model_data={
    'site': <Site: Some Site>,
    'tenant': None,
    'name': 'device02-2',
    'rack': None,
    'position': None,
    'face': '',
    'vc_position': 2,
    'vc_priority': None,
    'device_type': <DeviceType: WS-C3850-48P>,
    'device_role': <DeviceRole: some--role>,
    'platform': <Platform: ios>,
    'serial': 'FO??????YN',
    'asset_tag': None,
    'cluster': None,
    'virtual_chassis': <VirtualChassis: device02>
  }

Use Case

The all-in-one-line structured logs output by the tool are not especially readable, especially when unique ids involve a large number of distinct keys.

Issue with IPAM Parent/Child Prefixes

Environment

• Python version: 3.8.5
• Nautobot version: git branch develop @ 24d4ec371fb10c6ac0d686dc9578284c8e9cf04f
• nautobot-netbox-importer version: git branch gfm-error-handler @ 29a1031

Expected Behavior

Observed Behavior

Some prefixes were mentioned relatively early on with the log message Still unable to resolve reference? Later on those same prefixes were again referenced, this time with the message: Apparent duplicate object encountered...

For example:
21:25:37 Still unable to resolve reference? pk=1 source=ipaddress "62.67.209.64/31__{'pk': 1}__None__None__None" target=vrf
and then
21:25:50 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=prefix "62.67.209.64/31__None__{'name': 'Itenos'}__{'name': 'Alarislabs Pte Ltd'}" model_id=62.67.209.64/31__None__{'name': 'Itenos'}__{'name': 'Alarislabs Pte Ltd'}

Eventually it ended up crashing completely with one of the aforementioned prefixes logged as ...already present.

Here's the full trace:

Traceback (most recent call last):
  File "/opt/nautobot/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/opt/nautobot/lib/python3.8/site-packages/nautobot/core/cli.py", line 55, in main
    run_app(
  File "/opt/nautobot/lib/python3.8/site-packages/nautobot/core/runner/runner.py", line 268, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/opt/nautobot/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/opt/nautobot/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/nautobot/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/nautobot/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/opt/nautobot/lib/python3.8/site-packages/nautobot_netbox_importer/management/commands/import_netbox_json.py", line 92, in handle
    target.sync_from(source, flags=DiffSyncFlags.SKIP_UNMATCHED_DST)
  File "/opt/nautobot/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 314, in sync_from
    return super().sync_from(source, diff_class=diff_class, flags=flags)
  File "/opt/nautobot/lib/python3.8/site-packages/diffsync/__init__.py", line 464, in sync_from
    result = syncer.perform_sync()
  File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 251, in perform_sync
    changed |= self.sync_diff_element(element)
  File "/opt/nautobot/lib/python3.8/site-packages/diffsync/helpers.py", line 293, in sync_diff_element
    self.dst_diffsync.add(model)
  File "/opt/nautobot/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 230, in add
    super().add(obj)
  File "/opt/nautobot/lib/python3.8/site-packages/diffsync/__init__.py", line 623, in add
    raise ObjectAlreadyExists(f"Object {uid} already present")
diffsync.exceptions.ObjectAlreadyExists: Object 62.140.12.64/26__None__None__None already present

If it helps at all, this is what it looks like in Netbox:

Apparent duplicate object encountered (InventoryItem with parent None)

Environment

• Python version: 3.6.8
• Nautobot version: 1.0.0b2
• nautobot-netbox-importer version: 1.0.1

Expected Behavior

Import data from NetBox 2.10.7

Observed Behavior

14:37:35 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=inventoryitem "{'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'wdm5', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 2'}, 'name': 'Z2'}, 'position': 20, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': 'ABCD1234', 'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G" model_id={'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'wdm5', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 2'}, 'name': 'Z2'}, 'position': 20, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': 'ABCD1234', 'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G
14:37:35 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=inventoryitem "{'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'wdm5', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 2'}, 'name': 'Z2'}, 'position': 20, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': 'ABCD1234', 'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G" model_id={'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'wdm5', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 2'}, 'name': 'Z2'}, 'position': 20, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': 'ABCD1234', 'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G
14:37:35 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=inventoryitem "{'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'wdm6', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 2'}, 'name': 'Z2'}, 'position': 12, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': '', 'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G" model_id={'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'wdm6', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 2'}, 'name': 'Z2'}, 'position': 12, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': '', 'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G
14:37:35 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=inventoryitem "{'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'wdm6', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 2'}, 'name': 'Z2'}, 'position': 12, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': '', 'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G" model_id={'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'wdm6', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 2'}, 'name': 'Z2'}, 'position': 12, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': '', 'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G
14:37:35 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=inventoryitem "{'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'wdm7', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'Rack 1'}, 'position': 20, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': '', 
'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G" model_id={'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 
'wdm7', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'Rack 1'}, 'position': 20, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': '', 'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G
14:37:35 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=inventoryitem "{'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'wdm7', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'Rack 1'}, 'position': 20, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': '', 
'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G" model_id={'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 
'wdm7', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'Rack 1'}, 'position': 20, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': '', 'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G
14:37:35 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=inventoryitem "{'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'wdm8', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'Rack 1'}, 'position': 13, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': '', 
'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G" model_id={'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 
'wdm8', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'Rack 1'}, 'position': 13, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': '', 'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G
14:37:35 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=inventoryitem "{'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 'wdm8', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'Rack 1'}, 'position': 13, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': '', 
'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G" model_id={'site': {'name': 'Site'}, 'tenant': {'name': 'Tenant'}, 'name': 
'wdm8', 'rack': {'group': {'site': {'name': 'Site'}, 'name': 'Site Suite 1'}, 'name': 'Rack 1'}, 'position': 13, 'face': 'front', 'vc_position': None, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Infinera'}, 'model': 'TM-301/II'}, 'device_role': {'name': 'xWDM'}, 'platform': None, 'serial': '', 'asset_tag': None, 'cluster': None}__None__TPMRHEX-L/16G

Steps to Reproduce

1. $ sudo -u postgres psql
2. postgres=# DROP DATABASE nautobot;
3. postgres=# CREATE DATABASE nautobot;
4. postgres=# GRANT ALL PRIVILEGES ON DATABASE nautobot TO nautobot;
5. $ sudo systemctl restart nautobot nautobot-worker
6. $ nautobot-server migrate
7. $ nautobot-server createsuperuser
8. $ nautobot-server collectstatic
9. $ nautobot-server check
10. $ python netbox/manage.py dumpdata --traceback --format=json --exclude admin.logentry --exclude sessions.session --exclude extras.ObjectChange --exclude extras.Script --exclude extras.Report > /tmp/netbox_data.json
11. $ nautobot-server import_netbox_json /tmp/netbox_data.json 2.10.5

DOCS Minimum required version 2.10.3

Docs state "The plugin is compatible with Nautobot 1.0.0b3 and later and can handle JSON data exported from NetBox 2.10.x at present."
however nautobot-server import_netbox_json fails with "CommandError: Minimum NetBox version supported is 2.10.3"

`Missing required object` message

Environment

nautobot==1.0.0b2
nautobot-netbox-importer @ file:///opt/nautobot/nautobot-plugin-netbox-importer (develop branch as of 29-Mar-2021)

Expected Behavior

No log messages

Observed Behavior

Following Log messages

11:53:16 Nautobot reported an error about a missing required object action=update model=<class 'nautobot.dcim.models.devices.Device'> model_data={'site': <Site: Some Site>, 'tenant': None, 'name': 'device02-2', 'rack': None, 'position': None, 'face': '', 'vc_position': 2, 'vc_priority': None, 'device_type': <DeviceType: WS-C3850-48P>, 'device_role': <DeviceRole: some--role>, 'platform': <Platform: ios>, 'serial': 'FO??????YN', 'asset_tag': None, 'cluster': None, 'virtual_chassis': <VirtualChassis: device02>}
Device matching query does not exist.
11:53:16 device update did not return the model object. action=update diffs={'-': {'virtual_chassis': None}, '+': {'virtual_chassis': {'master': {'site': {'name': 'Some Site'}, 'tenant': None, 'name': 'device02', 'rack': None, 'position': None, 'face': '', 'vc_position': 1, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Cisco'}, 'model': 'WS-C3850-48P'}, 'device_role': {'name': 'some--role'}, 'platform': {'name': 'ios'}, 'serial': 'FOC????U0XG', 'asset_tag': None, 'cluster': None}, 'name': 'device02', 'domain': ''}}} dst=<NautobotDiffSync> flags=<DiffSyncFlags.SKIP_UNMATCHED_DST: 4> model=device src=<NetBox210DiffSync> status=failure unique_id={'name': 'Some Site'}__None__device02-2__None__None____2__None__{'manufacturer': {'name': 'Cisco'}, 'model': 'WS-C3850-48P'}__{'name': 'some--role'}__{'name': 'ios'}__FOC?????YN__None__None
11:53:16 No object resulted from sync, will not process child objects. action=update diffs={'-': {'virtual_chassis': None}, '+': {'virtual_chassis': {'master': {'site': {'name': 'Some Site'}, 'tenant': None, 'name': 'device02', 'rack': None, 'position': None, 'face': '', 'vc_position': 1, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Cisco'}, 'model': 'WS-C3850-48P'}, 'device_role': {'name': 'some--role'}, 'platform': {'name': 'ios'}, 'serial': 'FOC????U0XG', 'asset_tag': None, 'cluster': None}, 'name': 'device02', 'domain': ''}}} dst=<NautobotDiffSync> flags=<DiffSyncFlags.SKIP_UNMATCHED_DST: 4> model=device src=<NetBox210DiffSync> unique_id={'name': 'Some Site'}__None__device02-2__None__None____2__None__{'manufacturer': {'name': 'Cisco'}, 'model': 'WS-C3850-48P'}__{'name': 'some--role'}__{'name': 'ios'}__FOC?????YN__None__None
11:53:16 Nautobot reported an error about a missing required object action=update model=<class 'nautobot.dcim.models.devices.Device'> model_data={'site': <Site: Some Site>, 'tenant': None, 'name': 'device02-3', 'rack': None, 'position': None, 'face': '', 'vc_position': 3, 'vc_priority': None, 'device_type': <DeviceType: WS-C3850-48P>, 'device_role': <DeviceRole: some--role>, 'platform': <Platform: ios>, 'serial': 'FOC????U0WT', 'asset_tag': None, 'cluster': None, 'virtual_chassis': <VirtualChassis: device02>}
Device matching query does not exist.
11:53:16 device update did not return the model object. action=update diffs={'-': {'virtual_chassis': None}, '+': {'virtual_chassis': {'master': {'site': {'name': 'Some Site'}, 'tenant': None, 'name': 'device02', 'rack': None, 'position': None, 'face': '', 'vc_position': 1, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Cisco'}, 'model': 'WS-C3850-48P'}, 'device_role': {'name': 'some--role'}, 'platform': {'name': 'ios'}, 'serial': 'FOC????U0XG', 'asset_tag': None, 'cluster': None}, 'name': 'device02', 'domain': ''}}} dst=<NautobotDiffSync> flags=<DiffSyncFlags.SKIP_UNMATCHED_DST: 4> model=device src=<NetBox210DiffSync> status=failure unique_id={'name': 'Some Site'}__None__device02-3__None__None____3__None__{'manufacturer': {'name': 'Cisco'}, 'model': 'WS-C3850-48P'}__{'name': 'some--role'}__{'name': 'ios'}__FOC??????T__None__None
11:53:16 No object resulted from sync, will not process child objects. action=update diffs={'-': {'virtual_chassis': None}, '+': {'virtual_chassis': {'master': {'site': {'name': 'Some Site'}, 'tenant': None, 'name': 'device02', 'rack': None, 'position': None, 'face': '', 'vc_position': 1, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Cisco'}, 'model': 'WS-C3850-48P'}, 'device_role': {'name': 'some--role'}, 'platform': {'name': 'ios'}, 'serial': 'FOC2????XG', 'asset_tag': None, 'cluster': None}, 'name': 'device02', 'domain': ''}}} dst=<NautobotDiffSync> flags=<DiffSyncFlags.SKIP_UNMATCHED_DST: 4> model=device src=<NetBox210DiffSync> unique_id={'name': 'Some Site'}__None__device02-3__None__None____3__None__{'manufacturer': {'name': 'Cisco'}, 'model': 'WS-C3850-48P'}__{'name': 'some--role'}__{'name': 'ios'}__FOC???????T__None__None
11:53:16 Nautobot reported an error about a missing required object action=update model=<class 'nautobot.dcim.models.devices.Device'> model_data={'site': <Site: Some Site>, 'tenant': None, 'name': 'device02-4', 'rack': None, 'position': None, 'face': '', 'vc_position': 4, 'vc_priority': None, 'device_type': <DeviceType: WS-C3850-48P>, 'device_role': <DeviceRole: some--role>, 'platform': <Platform: ios>, 'serial': 'FCW????D0SP', 'asset_tag': None, 'cluster': None, 'virtual_chassis': <VirtualChassis: device02>}
Device matching query does not exist.
11:53:16 device update did not return the model object. action=update diffs={'-': {'virtual_chassis': None}, '+': {'virtual_chassis': {'master': {'site': {'name': 'Some Site'}, 'tenant': None, 'name': 'device02', 'rack': None, 'position': None, 'face': '', 'vc_position': 1, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Cisco'}, 'model': 'WS-C3850-48P'}, 'device_role': {'name': 'some--role'}, 'platform': {'name': 'ios'}, 'serial': 'FOC2214U0XG', 'asset_tag': None, 'cluster': None}, 'name': 'device02', 'domain': ''}}} dst=<NautobotDiffSync> flags=<DiffSyncFlags.SKIP_UNMATCHED_DST: 4> model=device src=<NetBox210DiffSync> status=failure unique_id={'name': 'Some Site'}__None__device02-4__None__None____4__None__{'manufacturer': {'name': 'Cisco'}, 'model': 'WS-C3850-48P'}__{'name': 'some--role'}__{'name': 'ios'}__FCW??????P__None__None
11:53:16 No object resulted from sync, will not process child objects. action=update diffs={'-': {'virtual_chassis': None}, '+': {'virtual_chassis': {'master': {'site': {'name': 'Some Site'}, 'tenant': None, 'name': 'device02', 'rack': None, 'position': None, 'face': '', 'vc_position': 1, 'vc_priority': None, 'device_type': {'manufacturer': {'name': 'Cisco'}, 'model': 'WS-C3850-48P'}, 'device_role': {'name': 'some--role'}, 'platform': {'name': 'ios'}, 'serial': 'FOC2?????XG', 'asset_tag': None, 'cluster': None}, 'name': 'device02', 'domain': ''}}} dst=<NautobotDiffSync> flags=<DiffSyncFlags.SKIP_UNMATCHED_DST: 4> model=device src=<NetBox210DiffSync> unique_id={'name': 'Some Site'}__None__device02-4__None__None____4__None__{'manufacturer': {'name': 'Cisco'}, 'model': 'WS-C3850-48P'}__{'name': 'some--role'}__{'name': 'ios'}__FC???????P__None__None

Steps to Reproduce

Apparent duplicate object encountered (InventoryItem with parent None)

Environment

nautobot==1.0.0b2
nautobot-netbox-importer @ file:///opt/nautobot/nautobot-plugin-netbox-importer (develop branch as of 29-Mar-2021)

Expected Behavior

No log warning for following different InventoryItem objects (different serial number field):

('device1', 'PVDM3 DSP DIMM with 256 Channels on Slot 0 SubSlot', '', 'Cisco', 'PVDM3-256', 'FOC????1111', None, True, 'PVDM3 DSP DIMM with 256 Channels')
('device1', 'PVDM3 DSP DIMM with 256 Channels on Slot 0 SubSlot', '', 'Cisco', 'PVDM3-256', 'FOC????10Y3', None, True, 'PVDM3 DSP DIMM with 256 Channels')

Observed Behavior

Log warning message appears for the different InventoryItems - they indeed are similar, but differ in the Serial Number Field:

10:56:46 Apparent duplicate object encountered? This may be an issue with your source data or may reflect a bug in this plugin. duplicate_id={'...' <TRUNCATED> pk_1=9123 pk_2=9124

Steps to Reproduce

1. Create two InventoryItems on a single device
2. Use the same Name, Part-ID, Manufacturer, Description
3. Use different serial number

Add support for importing NetBox 2.11.x data

Environment

• Nautobot version: 1.0.1
• nautobot-netbox-importer version: 1.4.0

Proposed Functionality

Get support for an exported json file of Netbox 2.11.x

Use Case

Currently is is not working when setting the version to 2.10.8

(nautobot) nautobot@debian:~$ nautobot-server import_netbox_json netbox_120721.json 2.10.8
05:42:36 info     Loading NetBox JSON data into memory...
  filename: {'netbox_120721.json'}
Traceback (most recent call last):
  File "/opt/nautobot/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/opt/nautobot/lib/python3.7/site-packages/nautobot/core/cli.py", line 62, in main
    initializer=_configure_settings,  # Called after defaults
  File "/opt/nautobot/lib/python3.7/site-packages/nautobot/core/runner/runner.py", line 266, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/opt/nautobot/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/opt/nautobot/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/nautobot/lib/python3.7/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/nautobot/lib/python3.7/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/opt/nautobot/lib/python3.7/site-packages/nautobot_netbox_importer/management/commands/import_netbox_json.py", line 40, in handle
    data = json.load(options["json_file"])
  File "/usr/lib/python3.7/json/__init__.py", line 296, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 3016066 (char 3016065)

Import Data That Does Not Meet Custom Field Validation

Environment

• Nautobot version: 1.0.0b2
• nautobot-netbox-importer version: 1.0.1

Proposed Functionality

Support importing data that does not conform to the latest constraints applied to a model using custom_fields. A current manual workaround is to set the custom_field as required: false to remove the validation check when importing data.

Use Case

Not all objects will have custom fields applied for a particular model. Custom fields can be configured to be required without retroactively applying that constraint against all objects for a model in netbox. When migrating to nautobot, the importer applies the custom_field requirement constraints against the data causing multiple data validation errors.

Traceback during sync phase when trying to import from Netbox 2.10.8 to Nautobot 1.2.3

Environment

• Python version: 2.10.8
• Nautobot version: 1.2.3
• nautobot-netbox-importer version: 1.4.0

Expected Behavior

Successful import

Observed Behavior

Importer traceback during import "sync" phase:

[root@ladm545l nautobot-docker-compose]# docker-compose exec -T nautobot sh -c "nautobot-server import_netbox_json --bypass-data-validation -v 3 /tmp/netbox_data.json 2.10.8"
15:05:06 info Loading NetBox JSON data into memory...
filename: {'/tmp/netbox_data.json'}
15:05:08 info JSON data loaded into memory successfully.
15:05:08 info Loading imported NetBox source data into DiffSync...
contenttype : 100%|██████████| 90/ 90 [00:00]
permission : 0%| | 0/ 363 [00:00]15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: admin
model: logentry
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: admin
model: logentry
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: admin
model: logentry
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: auth
model: permission
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: auth
model: permission
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: auth
model: permission
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: contenttypes
model: contenttype
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: contenttypes
model: contenttype
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: contenttypes
model: contenttype
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: sessions
model: session
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: sessions
model: session
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: sessions
model: session
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: useraction
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: useraction
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: useraction
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: reportresult
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: reportresult
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: reportresult
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: customfieldvalue
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: customfieldvalue
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: customfieldvalue
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: topologymap
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: topologymap
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: topologymap
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: graph
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: graph
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: graph
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: sessionkey
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: sessionkey
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: sessionkey
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: userkey
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: userkey
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: userkey
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: userkey
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: secret
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: secret
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: secret
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: secretrole
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: secretrole
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: secretrole
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: admin
model: logentry
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: auth
model: permission
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: contenttypes
model: contenttype
permission : 54%|█████▍ | 197/ 363 [00:00]15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: sessions
model: session
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: customfieldvalue
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: objectchange
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: objectchange
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: objectchange
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: objectchange
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: reportresult
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: graph
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: topologymap
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: secretrole
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: secret
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: userkey
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: secrets
model: sessionkey
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: job
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: job
15:05:08 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: job
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: job
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: job
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: users
model: userconfig
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: users
model: userconfig
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: users
model: userconfig
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: users
model: userconfig
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: job
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: job
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: job
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: extras
model: job
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: users
model: admingroup
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: users
model: admingroup
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: users
model: admingroup
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: users
model: admingroup
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: users
model: adminuser
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: users
model: adminuser
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: users
model: adminuser
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: users
model: adminuser
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: dcim
model: cablepath
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: dcim
model: cablepath
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: dcim
model: cablepath
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: dcim
model: cablepath
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: nextbox_ui_plugin
model: savedtopology
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: nextbox_ui_plugin
model: savedtopology
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: nextbox_ui_plugin
model: savedtopology
15:05:09 debug Flagging permission for extraneous content-type as ignorable
diffsync: NetBox210DiffSync
app_label: nextbox_ui_plugin
model: savedtopology
permission : 100%|██████████| 363/ 363 [00:00]
group : 100%|██████████| 13/ 13 [00:00]
user : 0%| | 0/ 2 [00:00]15:05:09 debug Looking for UserConfig corresponding to User
username: admin
15:05:09 debug Looking for UserConfig corresponding to User
username: xsicardj
user : 100%|██████████| 2/ 2 [00:00]
token : 100%|██████████| 1/ 1 [00:00]
customfield : 100%|██████████| 11/ 11 [00:00]
tenantgroup : 100%|██████████| 8/ 8 [00:00]
tenant : 100%|██████████| 61/ 61 [00:00]
region : 100%|██████████| 7/ 7 [00:00]
site : 100%|██████████| 18/ 18 [00:00]
manufacturer : 100%|██████████| 38/ 38 [00:00]
devicetype : 100%|██████████| 337/ 337 [00:00]
devicerole : 100%|██████████| 29/ 29 [00:00]
platform : 100%|██████████| 11/ 11 [00:00]
clustertype : 100%|██████████| 8/ 8 [00:00]
clustergroup : 100%|██████████| 8/ 8 [00:00]
cluster : 100%|██████████| 10/ 10 [00:00]
provider : 100%|██████████| 13/ 13 [00:00]
circuittype : 100%|██████████| 7/ 7 [00:00]
circuit : 100%|██████████| 52/ 52 [00:00]
circuittermination : 100%|██████████| 75/ 75 [00:00]
rackgroup : 100%|██████████| 275/ 275 [00:00]
rackrole : 100%|██████████| 16/ 16 [00:00]
rack : 100%|██████████| 651/ 651 [00:00]
rir : 100%|██████████| 12/ 12 [00:00]
aggregate : 100%|██████████| 69/ 69 [00:00]
role : 100%|██████████| 9/ 9 [00:00]
vlangroup : 100%|██████████| 124/ 124 [00:00]
vlan : 100%|██████████| 2565/ 2565 [00:00]
prefix : 100%|██████████| 5727/ 5727 [00:01]
device : 100%|██████████| 6671/ 6671 [00:01]
consoleporttemplate : 100%|██████████| 17/ 17 [00:00]
consoleserverporttemplate: 100%|██████████| 112/ 112 [00:00]
powerporttemplate : 100%|██████████| 30/ 30 [00:00]
poweroutlettemplate : 100%|██████████| 26/ 26 [00:00]
rearporttemplate : 100%|██████████| 253/ 253 [00:00]
frontporttemplate : 100%|██████████| 274/ 274 [00:00]
interfacetemplate : 100%|██████████| 454/ 454 [00:00]
devicebaytemplate : 100%|██████████| 170/ 170 [00:00]
devicebay : 100%|██████████| 1068/ 1068 [00:00]
virtualchassis : 100%|██████████| 5/ 5 [00:00]
virtualmachine : 100%|██████████| 7/ 7 [00:00]
consoleport : 100%|██████████| 109/ 109 [00:00]
consoleserverport : 100%|██████████| 32/ 32 [00:00]
powerport : 100%|██████████| 200/ 200 [00:00]
rearport : 100%|██████████| 2247/ 2247 [00:00]
frontport : 100%|██████████| 2444/ 2444 [00:00]
interface : 100%|██████████| 103765/103765 [00:18]
vminterface : 100%|██████████| 16/ 16 [00:00]
ipaddress : 100%|██████████| 12163/ 12163 [00:01]
cable : 100%|██████████| 61/ 61 [00:00]
tag : 100%|██████████| 61/ 61 [00:00]
configcontext : 100%|██████████| 1/ 1 [00:00]
exporttemplate : 100%|██████████| 1/ 1 [00:00]
webhook : 100%|██████████| 3/ 3 [00:00]
taggeditem : 100%|██████████| 28006/ 28006 [00:05]
15:05:40 info Data loading from NetBox source data complete.
15:05:40 info Loading data from Nautobot into DiffSync...
contenttype : 100%|██████████| 122/ 122 [00:00]
permission : 0%| | 0/ 488 [00:00]15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: admin
model: logentry
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: admin
model: logentry
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: admin
model: logentry
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: admin
model: logentry
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: auth
model: permission
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: auth
model: permission
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: auth
model: permission
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: auth
model: permission
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: contenttypes
model: contenttype
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: contenttypes
model: contenttype
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: contenttypes
model: contenttype
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: contenttypes
model: contenttype
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: database
model: constance
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: database
model: constance
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: database
model: constance
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: database
model: constance
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: dcim
model: cablepath
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: dcim
model: cablepath
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: dcim
model: cablepath
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: dcim
model: cablepath
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: clockedschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: clockedschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: clockedschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: clockedschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: crontabschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: crontabschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: crontabschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: crontabschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: intervalschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: intervalschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: intervalschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: intervalschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: periodictask
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: periodictask
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: periodictask
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: periodictask
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: periodictasks
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: periodictasks
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: periodictasks
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: periodictasks
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: solarschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: solarschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: solarschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: django_celery_beat
model: solarschedule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: computedfield
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: computedfield
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: computedfield
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: computedfield
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: configcontextschema
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: configcontextschema
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: configcontextschema
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: configcontextschema
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: fileattachment
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: fileattachment
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: fileattachment
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: fileattachment
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: fileproxy
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: fileproxy
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: fileproxy
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: fileproxy
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: gitrepository
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: gitrepository
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: gitrepository
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: gitrepository
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: graphqlquery
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: graphqlquery
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: graphqlquery
permission : 48%|████▊ | 235/ 488 [00:00]15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: graphqlquery
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: healthchecktestmodel
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: healthchecktestmodel
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: healthchecktestmodel
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: healthchecktestmodel
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: job
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: job
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: job
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: job
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: joblogentry
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: joblogentry
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: joblogentry
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: joblogentry
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: objectchange
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: objectchange
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: objectchange
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: objectchange
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: relationship
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: relationship
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: relationship
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: relationship
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: relationshipassociation
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: relationshipassociation
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: relationshipassociation
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: relationshipassociation
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: scheduledjob
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: scheduledjob
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: scheduledjob
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: scheduledjob
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: scheduledjobs
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: scheduledjobs
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: scheduledjobs
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: scheduledjobs
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: secret
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: secret
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: secret
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: secret
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: secretsgroup
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: secretsgroup
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: secretsgroup
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: secretsgroup
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: secretsgroupassociation
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: secretsgroupassociation
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: secretsgroupassociation
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: secretsgroupassociation
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: status
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: status
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: status
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: extras
model: status
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: circuitimpact
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: circuitimpact
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: circuitimpact
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: circuitimpact
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: circuitmaintenance
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: circuitmaintenance
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: circuitmaintenance
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: circuitmaintenance
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: note
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: note
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: note
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: note
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: notificationsource
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: notificationsource
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: notificationsource
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: notificationsource
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: parsednotification
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: parsednotification
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: parsednotification
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: parsednotification
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: rawnotification
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: rawnotification
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: rawnotification
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_circuit_maintenance
model: rawnotification
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_data_validation_engine
model: minmaxvalidationrule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_data_validation_engine
model: minmaxvalidationrule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_data_validation_engine
model: minmaxvalidationrule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_data_validation_engine
model: minmaxvalidationrule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_data_validation_engine
model: regularexpressionvalidationrule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_data_validation_engine
model: regularexpressionvalidationrule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_data_validation_engine
model: regularexpressionvalidationrule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_data_validation_engine
model: regularexpressionvalidationrule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_device_onboarding
model: onboardingdevice
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_device_onboarding
model: onboardingdevice
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_device_onboarding
model: onboardingdevice
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_device_onboarding
model: onboardingdevice
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_device_onboarding
model: onboardingtask
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_device_onboarding
model: onboardingtask
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_device_onboarding
model: onboardingtask
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_device_onboarding
model: onboardingtask
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: compliancefeature
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: compliancefeature
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: compliancefeature
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: compliancefeature
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: compliancerule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: compliancerule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: compliancerule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: compliancerule
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: configcompliance
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: configcompliance
15:05:40 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: configcompliance
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: configcompliance
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: configremove
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: configremove
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: configremove
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: configremove
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: configreplace
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: configreplace
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: configreplace
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: configreplace
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: goldenconfig
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: goldenconfig
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: goldenconfig
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: goldenconfig
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: goldenconfigsetting
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: goldenconfigsetting
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: goldenconfigsetting
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: nautobot_golden_config
model: goldenconfigsetting
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: sessions
model: session
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: sessions
model: session
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: sessions
model: session
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: sessions
model: session
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: association
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: association
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: association
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: association
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: code
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: code
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: code
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: code
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: nonce
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: nonce
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: nonce
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: nonce
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: partial
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: partial
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: partial
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: partial
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: usersocialauth
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: usersocialauth
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: usersocialauth
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: social_django
model: usersocialauth
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: users
model: admingroup
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: users
model: admingroup
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: users
model: admingroup
15:05:41 debug Flagging permission for extraneous content-type as ignorable
diffsync: NautobotDiffSync
app_label: users
model: admingroup
permission : 100%|██████████| 488/ 488 [00:00]
status : 100%|██████████| 19/ 19 [00:00]
group : 100%|██████████| 13/ 13 [00:00]
user : 100%|██████████| 49/ 49 [00:00]
token : 100%|██████████| 11/ 11 [00:00]
customfield : 100%|██████████| 19/ 19 [00:00]
15:05:41 info Data loading from Nautobot complete.
15:05:41 info Beginning data synchronization...
| | 0/ ? [00:00]15:05:41 info Beginning diff calculation
src: NetBox210DiffSync
dst: NautobotDiffSync
flags: DiffSyncFlags.SKIP_UNMATCHED_DST
diff: 99%|█████████▉| 167854/169527 [00:30]15:06:12 info Diff calculation complete
src: NetBox210DiffSync
dst: NautobotDiffSync
flags: DiffSyncFlags.SKIP_UNMATCHED_DST
15:06:12 info Beginning sync
src: NetBox210DiffSync
dst: NautobotDiffSync
flags: DiffSyncFlags.SKIP_UNMATCHED_DST
diff: 100%|██████████| 169527/169527 [00:30]
sync: 0%| | 0/168353 [00:00]15:06:12 info Updated successfully
src: NetBox210DiffSync
dst: NautobotDiffSync
flags: DiffSyncFlags.SKIP_UNMATCHED_DST
action: update
model: customfield
unique_id: 54ace47c-0e04-595c-9833-afab5bdff6e3
diffs:
{'+': {'content_types': [{'app_label': 'ipam', 'model': 'prefix'}]},
'-': {'content_types': []}}
status: success
sync: 0%| | 23/168353 [00:00]
Traceback (most recent call last):
File "/usr/local/bin/nautobot-server", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/nautobot/core/cli.py", line 54, in main
run_app(
File "/usr/local/lib/python3.9/site-packages/nautobot/core/runner/runner.py", line 266, in run_app
management.execute_from_command_line([runner_name, command] + command_args)
File "/usr/local/lib/python3.9/site-packages/django/core/management/init.py", line 401, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.9/site-packages/django/core/management/init.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 330, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 371, in execute
output = self.handle(*args, **options)
File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_netbox_importer/management/commands/import_netbox_json.py", line 66, in handle
target.sync_from(source, flags=DiffSyncFlags.SKIP_UNMATCHED_DST, callback=p_bar.diffsync_callback)
File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 300, in sync_from
return super().sync_from(source, diff_class=diff_class, flags=flags, callback=callback)
File "/opt/nautobot/.local/lib/python3.9/site-packages/diffsync/init.py", line 476, in sync_from
result = syncer.perform_sync()
File "/opt/nautobot/.local/lib/python3.9/site-packages/diffsync/helpers.py", line 313, in perform_sync
changed |= self.sync_diff_element(element)
File "/opt/nautobot/.local/lib/python3.9/site-packages/diffsync/helpers.py", line 345, in sync_diff_element
changed, modified_model = self.sync_model(model, ids, attrs)
File "/opt/nautobot/.local/lib/python3.9/site-packages/diffsync/helpers.py", line 393, in sync_model
model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs)
File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 295, in create
record = cls.create_nautobot_record(
File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 230, in create_nautobot_record
record.created = model_data["created"]
KeyError: 'created'

Steps to Reproduce

1. Export netbox data:
   /opt/netbox/venv/bin/python /opt/netbox/netbox/manage.py dumpdata --traceback --format=json --exclude admin.logentry --exclude sessions.session --e xclude extras.ObjectChange --exclude extras.Script --exclude extras.Report"
2. Copy netbox data to nautobot container
   docker cp /data/netbox-exports/netbox-data-export-20220118-0956.json "$(docker-compose ps -q nautobot)":/tmp/netbox_data.json
3. Run import within nautobot container
   docker-compose exec -T nautobot sh -c "nautobot-server import_netbox_json --bypass-data-validation -v 3 /tmp/netbox_data.json 2.10.8"

Add support for importing NetBox 2.10.10 data

Environment

• Nautobot version:
• nautobot-netbox-importer version:

Proposed Functionality

Add the ability to import from NetBox 2.10.10.

Use Case

Allow the import of NetBox 2.10.10 data to allow easy migration from NetBox 2.10.10.

I have a NetBox 2.10.10 instance with a lot of data in it I would like to migrate to NautoBot. It would be epic if this could be done.
Thank you!

Duplicate Object Reports Should Include Primary Key

Environment

• Python version: 3.8.5
• Nautobot version: 1.0.0b2
• nautobot-netbox-importer version: 1.0.1
• Container: nautobot-lab

Expected Behavior

Duplicate object report includes an objects primary key to aid troubleshooting.

Observed Behavior

Duplicate object reports model and model_id.

12:54:58 Apparent duplicate object encountered. This may be an issue with your source data or may reflect a bug in this plugin. model=vlangroup "None__XX" model_id=None__XX

Steps to Reproduce

1. nautobot-server import_netbox_json data.json 2.10.3

Import ObjectChange objects

Proposed Functionality

Add the capacity to, under demand, import ObjectChange objects from Netbox to Nautobot.

Use Case

This would enable to keep change log records.

Suppress warning errors by default

Environment

• Nautobot version: 1.0.2
• nautobot-netbox-importer version: 1.0.3

Proposed Functionality

Would be nice to suppress by default all warning errors in stdout. Maybe add a flag --warning to enable warning messages or even better -v (Ansible like) to return more verbose output