h4ndzdatm0ld/nautobot-ssot-vsphere

Nautobot SSoT VMWare vSphere

nautobot-ssot-vsphere's Issues

An exception occurred: KeyError: 'disks'

Environment

• Python version: 3.9
• Nautobot version: 1.4.5
• nautobot-ssot-vsphere version: 0.1.3

Expected Behavior

Importing my VMs

Observed Behavior

An exception occured

Steps to Reproduce

1. Set up Nautobot
2. Import VMs
3. Profit

Detailed error Message:

Traceback (most recent call last):
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_ssot/jobs/base.py", line 332, in run
    self.sync_data()
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_ssot_vsphere/jobs.py", line 146, in sync_data
    vsphere_source.load()
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 223, in load
    self.load_data()
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 195, in load_data
    self.load_virtualmachines(cluster, diffsync_cluster)
  File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 61, in load_virtualmachines
    "disk": get_disk_total(virtual_machine_details["disks"]),
KeyError: 'disks'

Since I am not really sure which VM is affected, its hard finding out, since we have a few hundred of them.
Curling the API for a few VMs for the disks succeeded.

ObjectNotFound: diffsync_cluster XXX not present in LocalStore

Environment

• Python version: 3.10.4
• Nautobot version: 1.4.2
• nautobot-ssot-vsphere version: 0.1.2

Expected Behavior

After i added another Cluster and filled it with data, i re-run the sync with the nautobot-ssot-vsphere plugin. It should ignore the manually added Cluster and sync the data between the configured vSphere Cluster and nautobot.

Observed Behavior

nautobot-ssot-vsphere prints out the following error:

Traceback (most recent call last): File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot/jobs/base.py", line 327, in run self.sync_data() File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot_vsphere/jobs.py", line 156, in sync_data dest.load() File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_nautobot.py", line 201, in load self.load_data() File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_nautobot.py", line 197, in load_data self.load_virtual_machines() File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_nautobot.py", line 129, in load_virtual_machines diffsync_cluster = self.get(self.diffsync_cluster, virtual_machine.cluster.name) File "/opt/nautobot/lib/python3.10/site-packages/diffsync/__init__.py", line 639, in get return self.store.get(model=obj, identifier=identifier) File "/opt/nautobot/lib/python3.10/site-packages/diffsync/store/local.py", line 49, in get raise ObjectNotFound(f"{modelname} {uid} not present in {str(self)}") diffsync.exceptions.ObjectNotFound: diffsync_cluster **another_cluster** not present in LocalStore

Steps to Reproduce

1. Add another cluster and fill it with data (e.g., add a virtual machine)
2. Re-run the sync
3. Receive the error message

I tried also the following:

• using Tags only (same result)
• using Cluster filter (same result)

After each sync, manually added Clusters are deleted

Environment

• Python version: 3.10
• Nautobot version: 1.4.0
• nautobot-ssot-vsphere version: 0.1.3

Expected Behavior

We have multiple clusters from different vendors, but only the data from one vSphere Cluster is synced with nautobot. We expect that data which is not on the vSphere Cluster site is still available after the sync.

Observed Behavior

When the sync is done, every other Cluster is deleted.

Steps to Reproduce

1. Create additional Cluster "new Cluster"
2. Sync
3. "new Cluster" is gone

Multiple Clusters with same Name get created once

Environment

• Python version: 3.9
• Nautobot version: 1.4.5
• nautobot-ssot-vsphere version: 0.1.3

Expected Behavior

Creating all Clusters

Observed Behavior

If you have multiple clusters with the same name in vsphere, only one cluster in nautobot gets created and subsequently alle vms are assigned to this one cluster.

Steps to Reproduce

1. see above

My vSphere looks like this:

• vSphere-Server
  • My Site 1
    • Prod
  • My Site 2
    • Prod
  • My Site 3
    • Prod

So your plugin should create three cluster groups (My Site 1 to 3) and three clusters (Prod) associated with a cluster group each.
Instead it creates 3 cluster groups, one cluster "prod", associated with cluster group "My Site 3" and adds everything to this cluster.

Job fails with `TypeError: string indices must be integers`

Environment

• Python version: 3.8.12
• Nautobot version: 1.3.10
• nautobot-ssot-vsphere version: 0.2.5

Expected Behavior

Import runs (dry run)

Observed Behavior

Traceback (most recent call last):
  File "/opt/nautobot/lib64/python3.8/site-packages/nautobot_ssot/jobs/base.py", line 327, in run
    self.sync_data()
  File "/opt/nautobot/lib64/python3.8/site-packages/nautobot_ssot_vsphere/jobs.py", line 146, in sync_data
    vsphere_source.load()
  File "/opt/nautobot/lib64/python3.8/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 218, in load
    self.load_data()
  File "/opt/nautobot/lib64/python3.8/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 165, in load_data
    clustergroups = self.load_cluster_groups()
  File "/opt/nautobot/lib64/python3.8/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 45, in load_cluster_groups
    self.get_or_instantiate(self.diffsync_clustergroup, {"name": clustergroup["name"]})
TypeError: string indices must be integers

Steps to Reproduce

1. Run job (dry run)

Error if try to delete job result log

Environment

• Python version: 3.9.16
• Nautobot version: 1.5.11
• nautobot-ssot-vsphere version: 0.1.4

Expected Behavior

Delete job result log

Observed Behavior

<class 'django.urls.exceptions.NoReverseMatch'>

'nautobot_ssot' is not a registered namespace inside 'plugins'

Steps to Reproduce

1. cat nautobot_config.py
   ........
   EXTRA_INSTALLED_APPS = ["nautobot_ssot"]
   PLUGINS = ["nautobot_ssot"]
   PLUGINS = ["nautobot_ssot_vsphere"]
   PLUGINS_CONFIG = {
   "nautobot_ssot_vsphere": {
   "VSPHERE_URI": os.getenv("https://HOSTNAME.domain/"),
   "VSPHERE_USERNAME": os.getenv("USERNAME"),
   "VSPHERE_PASSWORD": os.getenv("PASSWORD"),
   "VSPHERE_VERIFY_SSL": is_truthy(os.getenv("VSPHERE_VERIFY_SSL", False)),
   },
   "nautobot_ssot": {
   "hide_example_jobs": True
   },
   }
2. Jobs - Last run - Job result - Delete -Confirm -

After installing nautobot-ssot-vsphere, nautobot is unable to start.

Environment

• Python version: 3.8
• Nautobot version: 1.4.7
• nautobot-ssot-vsphere version: 0.1.3

I have a running installation of nautobot. Then I added the plugin nautobot-ssot-vsphere, after restarted nautbot, it is unable to start and throws the next error: Model class nautobot_ssot.models.Sync doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.

The full error trace is attached with the name trace.txt

Steps to Reproduce

1. Fresh nautobot installation.
2. Install nautobot-ssot-vsphere python package
3. Configure nautobot_config.py with the required parameters.
4. Start nautubot.

My nautobot_config.py:

PLUGINS = ["nautobot_device_onboarding","nautobot_plugin_nornir","nautobot_golden_config","nautobot_ssot_vsphere"]

PLUGINS_CONFIG = {
    "nautobot_plugin_nornir": {
        "nornir_settings": {
            "credentials": "nautobot_plugin_nornir.plugins.credentials.env_vars.CredentialsEnvVars",
            "runner": {
                "plugin": "threaded",
                "options": {
                    "num_workers": 20,
                },
            },
        },
    },
    "nautobot_golden_config": {
        "per_feature_bar_width": 0.15,
        "per_feature_width": 13,
        "per_feature_height": 4,
        "enable_backup": True,
        "enable_compliance": True,
        "enable_intended": True,
        "enable_sotagg": True,
        "sot_agg_transposer": None,
        "enable_postprocessing": False,
        "postprocessing_callables": [],
        "postprocessing_subscribed": [],
        "platform_slug_map": None,
        # "get_custom_compliance": "my.custom_compliance.func"
    },
    "nautobot_ssot_vsphere": {
        "VSPHERE_URI": os.getenv("VSPHERE_URI"),
        "VSPHERE_USERNAME": os.getenv("VSPHERE_USERNAME"),
        "VSPHERE_PASSWORD": os.getenv("VSPHERE_PASSWORD"),
        "VSPHERE_VERIFY_SSL": is_truthy(os.getenv("VSPHERE_VERIFY_SSL", False)),
    },
}

trace.txt

When syncing with Tag filter, the job fails, even if the Sync itself completes. Reason: Deletion failed protected object

Environment

• Python version: 3.10.6
• Nautobot version: 1.4.5
• nautobot-ssot-vsphere version: 0.3.0 (origin/bug/23/diffsync-exception-cluster)

Expected Behavior

The Sync completes with status Success

Observed Behavior

The Sync fails with status Success

Steps to Reproduce

1. Create a SSOT vSphere Sync
2. Do a init Sync to nautobot
3. Create any other cluster in nautobot independent from the plugin and tag it differently
4. Sync with "Tag filtering"
5. The Sync fails with Status Success, giving the error message "Deletion failed protected object" for any other cluster which was created independent.

Cannot import VMs

Environment

• Python version: 3.10.6
• Nautobot version: 1.15
• nautobot-ssot-vsphere version: 0.1.4 |  

Expected Behavior

All of VMs hosted by VMware cluster are visible on the Nautobot

Observed Behavior

0 VMs synchronized

Steps to Reproduce

Hello

I am trying to synchronize the Nautobot with ssot vmware plugin and fetch all details about cluster/vms unfortunately basid configuration get only top cluster names without virtual machines. Example log:

2022-11-14 12:52:38.208543 run Info —
Starting job with the following options: Debug: True, Dry Run: False, Sync Tagged Only: False, Cluster Filter: None

2022-11-14 12:52:38.434979 run Info —
Loading current data from vSphere...

2022-11-14 12:52:38.691999 run Default —
Loading ClusterGroups [{'name': 'cluster_DC_N', 'datacenter': 'datacenter-137'}, {'name': 'cluster_DC_G', 'datacenter': 'datacenter-105'}, {'name': 'cluster_DC_A', 'datacenter': 'datacenter-65'}, {'name': 'cluster_DC_A2', 'datacenter': 'datacenter-2'}]

2022-11-14 12:52:38.712823 run Default —
Found vSphere Clusters []

2022-11-14 12:52:38.735944 run Default —
Found vSphere Clusters []

2022-11-14 12:52:38.754198 run Default —
Found vSphere Clusters []

2022-11-14 12:52:38.773489 run Default —
Found vSphere Clusters []

2022-11-14 12:52:38.776267 run Info —
Loading current data from Nautobot...

2022-11-14 12:52:38.780313 run Default —
Found 4 Nautobot ClusterGroup objects

2022-11-14 12:52:38.784956 run Default —
Found 0 Nautobot Cluster objects

2022-11-14 12:52:38.789069 run Default —
Found 0 Virtual Machine objects

2022-11-14 12:52:38.792319 run Info —
Calculating diffs...

2022-11-14 12:52:38.795644 run Default —
Diff: {}

2022-11-14 12:52:38.799734 run Info —
DiffSync Summary: Create: 0, Update: 0, Delete: 0, No Change: 4

2022-11-14 12:52:38.801842 run Info —
Syncing from vSphere to Nautobot

2022-11-14 12:52:38.806153 run Success —
Sync complete.

At least in my understanding it should;g be string forward to get all the VMs in nautobot. What I missed?

Model class nautobot_ssot.models.Sync doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS

Environment

• Python version: 3.9.2
• Nautobot version: 1.5.18
• nautobot-ssot-vsphere version: 0.1.4

nautobot service and worker service doesn't start

Observed Behavior

Traceback (most recent call last):
File "/scdata/nautobot/bin/nautobot-server", line 8, in
sys.exit(main())
File "/scdata/nautobot/lib/python3.9/site-packages/nautobot/core/cli.py", line 54, in main
run_app(
File "/scdata/nautobot/lib/python3.9/site-packages/nautobot/core/runner/runner.py", line 263, in run_app
management.execute_from_command_line([runner_name, command] + command_args)
File "/scdata/nautobot/lib/python3.9/site-packages/django/core/management/init.py", line 419, in execute_from_command_line
utility.execute()
File "/scdata/nautobot/lib/python3.9/site-packages/django/core/management/init.py", line 395, in execute
django.setup()
File "/scdata/nautobot/lib/python3.9/site-packages/django/init.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/scdata/nautobot/lib/python3.9/site-packages/django/apps/registry.py", line 122, in populate
app_config.ready()
File "/scdata/nautobot/lib/python3.9/site-packages/nautobot_ssot_vsphere/init.py", line 46, in ready
super().ready()
File "/scdata/nautobot/lib/python3.9/site-packages/nautobot/extras/plugins/init.py", line 149, in ready
jobs = import_object(f"{self.module}.{self.jobs}")
File "/scdata/nautobot/lib/python3.9/site-packages/nautobot/extras/plugins/utils.py", line 45, in import_object
spec.loader.exec_module(module)
File "", line 790, in exec_module
File "", line 228, in _call_with_frames_removed
File "/scdata/nautobot/lib/python3.9/site-packages/nautobot_ssot_vsphere/jobs.py", line 14, in
from nautobot_ssot.jobs.base import DataMapping, DataSource
File "/scdata/nautobot/lib/python3.9/site-packages/nautobot_ssot/jobs/init.py", line 7, in
from .base import DataSource, DataTarget
File "/scdata/nautobot/lib/python3.9/site-packages/nautobot_ssot/jobs/base.py", line 24, in
from nautobot_ssot.models import Sync, SyncLogEntry
File "/scdata/nautobot/lib/python3.9/site-packages/nautobot_ssot/models.py", line 39, in
class Sync(BaseModel):
File "/scdata/nautobot/lib/python3.9/site-packages/django/db/models/base.py", line 113, in new
raise RuntimeError(
RuntimeError: Model class nautobot_ssot.models.Sync doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.

Steps to Reproduce

install with last versions available

An exception occurred: KeyError: 'ip'

Environment

• Python version: 3.10
• Nautobot version: 1.3.6
• nautobot-ssot-vsphere version: 0.1.1

Expected Behavior

The Job is running fine for a while and should update the Inventory for the updated Infrastructure.

Observed Behavior

The scheduled job (also when i run it manually) creates this error message after a while:

Traceback (most recent call last):
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot/jobs/base.py", line 327, in run
    self.sync_data()
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot_vsphere/jobs.py", line 146, in sync_data
    vsphere_source.load()
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 220, in load
    self.load_data()
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 193, in load_data
    self.load_virtualmachines(cluster, diffsync_cluster)
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 64, in load_virtualmachines
    self.load_vm_interfaces(
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 154, in load_vm_interfaces
    ipv4_addresses, ipv6_addresses = self.load_ip_addresses(
  File "/opt/nautobot/lib/python3.10/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 84, in load_ip_addresses
    for ip_address in interface["ip"]["ip_addresses"]:
KeyError: 'ip'

Steps to Reproduce

1. Run the Job initially and schedule it
2. Wait a time (some days)
3. The Job will fail creating a diff-change with the Keyerror 'ip'

Option to filter IPv6 link-local addresses from being imported.

Environment

• Nautobot version: 1.3.10
• nautobot-ssot-vsphere version: 0.2.5

Proposed Functionality

Add an option to filter IPv6 link-local addresses. Nautobot-ssot-vsphere currently imports IPv6 link-local addresses (FE80::/10). These are local to the link, and don't really have much in terms of meaning in an IPAM/DCIM. Also, they can be duplicated across links (even on the same device), which Nautobot's model doesn't support.

Use Case

Error if run job task

Environment

• Python version: 3.9.16
• Nautobot version: 1.5.11
• nautobot-ssot-vsphere version: 0.1.4

Expected Behavior

Job run without errors

Observed Behavior

Job finished with error
ValueError: Missing required parameters for vSphere API access - check environment and plugin configuration

Steps to Reproduce

1.cat nautobot_config.py
........
PLUGINS = ["nautobot_ssot_vsphere", "nautobot_ssot"]
PLUGINS_CONFIG = {
"nautobot_ssot_vsphere": {
"VSPHERE_URI": os.getenv("https://hostname.domain/"),
"VSPHERE_USERNAME": os.getenv("USERNAME"),
"VSPHERE_PASSWORD": os.getenv("PASSWORD"),
"VSPHERE_VERIFY_SSL": is_truthy(os.getenv("VSPHERE_VERIFY_SSL", False)),
},
"nautobot_ssot": {
"hide_example_jobs": True
},
}
2. Job - Jobs - SSoT - Virtualization ([VMWare vSphere ⟹ Nautobot]) - Run

Error if run job task

Environment

• Python version: 3.9.16
• Nautobot version: 1.5.11
• nautobot-ssot-vsphere version: 0.1.4

Expected Behavior

Job run without errors

Observed Behavior

Job finished with error
An exception occurred: KeyError: 'value'

Traceback (most recent call last):
File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_ssot/jobs/base.py", line 332, in run
self.sync_data()
File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_ssot_vsphere/jobs.py", line 146, in sync_data
vsphere_source.load()
File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 227, in load
self.load_data()
File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 197, in load_data
self.load_virtualmachines(cluster, diffsync_cluster)
File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 69, in load_virtualmachines
self.load_vm_interfaces(
File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 158, in load_vm_interfaces
ipv4_addresses, ipv6_addresses = self.load_ip_addresses(
File "/opt/nautobot/.local/lib/python3.9/site-packages/nautobot_ssot_vsphere/diffsync/adapters/adapter_vsphere.py", line 82, in load_ip_addresses
for interface in vsphere_vm_interfaces.json()["value"]:
KeyError: 'value'

Step before failed

Steps to Reproduce

1. Fresh nautobot installation.
2. Install nautobot-ssot-vsphere python package
3. Run job
   Jobs - Jobs - SSoT - Virtualization ([VMWare vSphere ⟹ Nautobot]) - Run
   

Is this plugin still in development ? Fails to work with Nautobot v1.6.0

Environment

• Python version: Python 3.8.17
• Nautobot version: Nautobot v1.6.0
• nautobot-ssot-vsphere version: Release v0.1.4
• Linux distribution version: CentOS Stream 8

Expected Behavior

Install as described in instructions

Observed Behavior

Installation breaks wit warning about pip version, fails to create Job for vsphere discovery.

Tried to install plugin by following documentation:

pip install nautobot-ssot-vsphere

Install throws a error about pip:

[nautobot@Nautobot ~]$ pip install nautobot-ssot-vsphere
Requirement already satisfied: nautobot-ssot-vsphere in ./lib/python3.8/site-packages (0.1.4)
Requirement already satisfied: diffsync<2.0.0,>=1.4.3 in ./lib/python3.8/site-packages (from nautobot-ssot-vsphere) (1.8.0)
Requirement already satisfied: nautobot-ssot<2.0.0,>=1.1.0 in ./lib/python3.8/site-packages (from nautobot-ssot-vsphere) (1.3.2)
Requirement already satisfied: netutils<2.0.0,>=1.1.0 in ./lib/python3.8/site-packages (from nautobot-ssot-vsphere) (1.5.0)
Requirement already satisfied: colorama<0.5.0,>=0.4.3 in ./lib/python3.8/site-packages (from diffsync<2.0.0,>=1.4.3->nautobot-ssot-vsphere) (0.4.6)
Requirement already satisfied: packaging<24.0,>=21.3 in ./lib/python3.8/site-packages (from diffsync<2.0.0,>=1.4.3->nautobot-ssot-vsphere) (23.1)
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<2.0.0,>=1.7.4 in ./lib/python3.8/site-packages (from diffsync<2.0.0,>=1.4.3->nautobot-ssot-vsphere) (1.9.2)
Requirement already satisfied: structlog<23.0.0,>=20.1.0 in ./lib/python3.8/site-packages (from diffsync<2.0.0,>=1.4.3->nautobot-ssot-vsphere) (20.2.0)
Requirement already satisfied: Markdown!=3.3.5 in ./lib/python3.8/site-packages (from nautobot-ssot<2.0.0,>=1.1.0->nautobot-ssot-vsphere) (3.3.7)
Requirement already satisfied: prometheus-client<0.15.0,>=0.14.1 in ./lib/python3.8/site-packages (from nautobot-ssot<2.0.0,>=1.1.0->nautobot-ssot-vsphere) (0.14.1)
Requirement already satisfied: importlib-metadata>=4.4 in ./lib/python3.8/site-packages (from Markdown!=3.3.5->nautobot-ssot<2.0.0,>=1.1.0->nautobot-ssot-vsphere) (4.13.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./lib/python3.8/site-packages (from pydantic!=1.8,!=1.8.1,<2.0.0,>=1.7.4->diffsync<2.0.0,>=1.4.3->nautobot-ssot-vsphere) (3.10.0.2)
Requirement already satisfied: zipp>=0.5 in ./lib/python3.8/site-packages (from importlib-metadata>=4.4->Markdown!=3.3.5->nautobot-ssot<2.0.0,>=1.1.0->nautobot-ssot-vsphere) (3.16.2)
WARNING: There was an error checking the latest version of pip.

In Nautobot GUI, plugin is visible in installed plugins, but no Job is created for Vsphere discovery and can't be created manualy.