nautobot/nautobot-app-circuit-maintenance

Circuit Maintenance App for Nautobot

Nautobot

Nautobot is a Network Source of Truth and Network Automation Platform built as a web application atop the Django Python framework with a PostgreSQL or MySQL database.

Key Use Cases

1. Flexible Source of Truth for Networking - Nautobot core data models are used to define the intended state of network infrastructure enabling it as a Source of Truth. While a baseline set of models are provided (such as IP networks and addresses, devices and racks, circuits and cable, etc.) it is Nautobot's goal to offer maximum data model flexibility. This is enabled through features such as user-defined relationships, custom fields on any model, and data validation that permits users to codify everything from naming standards to having automated tests run before data can be populated into Nautobot.

2. Extensible Data Platform for Automation - Nautobot has a rich feature set to seamlessly integrate with network automation solutions. Nautobot offers GraphQL and native Git integration along with REST APIs and webhooks. Git integration dynamically loads YAML data files as Nautobot config contexts. Nautobot also has an evolving plugin system that enables users to create custom models, APIs, and UI elements. The plugin system is also used to unify and aggregate disparate data sources creating a Single Source of Truth to streamline data management for network automation.

3. Platform for Network Automation Apps - The Nautobot plugin system enables users to create Network Automation Apps. Apps can be as lightweight or robust as needed based on user needs. Using Nautobot for creating custom applications saves up to 70% development time by re-using features such as authentication, permissions, webhooks, GraphQL, change logging, etc. all while having access to the data already stored in Nautobot. Some production ready applications include:

The complete documentation for Nautobot can be found at Read the Docs.

Questions? Comments? Start by perusing our GitHub discussions for the topic you have in mind, or join the #nautobot channel on Network to Code's Slack community!

Build Status

Branch	Status
main
develop
next

Screenshots

---

---

---

---

Installation

Please see the documentation for instructions on installing Nautobot.

Application Stack

Below is a simplified overview of the Nautobot application stack for reference:

Plugins and Extensibility

Nautobot offers the ability to customize your setup to better align with your direct business needs. It does so through the use of various plugins that have been developed for network automation, and are designed to be used in environments where needed.

There are many plugins available within the Nautobot Apps ecosystem. The below screenshots are an example of some popular ones that are currently available.

Plugin Screenshots

Golden Config Plugin

ChatOps Plugin

Device Lifecycle Management Plugin

Providing Feedback

The best platform for general feedback, assistance, and other discussion is our GitHub discussions. To report a bug or request a specific feature, please open a GitHub issue using the appropriate template.

If you are interested in contributing to the development of Nautobot, please read our contributing guide prior to beginning any work.

Related projects

Please check out the GitHub nautobot topic for a list of relevant community projects.

Notices

Nautobot was initially developed as a fork of NetBox (v2.10.4). NetBox was originally developed by Jeremy Stretch at DigitalOcean and the NetBox Community.

nautobot-app-circuit-maintenance's Issues

"An error occurred in the current transaction" when encountering duplicate (?) notification

Environment

• Python version:
• Nautobot version: 1.1.2
• nautobot-circuit-maintenance version: 0.2.3

Expected Behavior

Circuit maintenance job to run to completion and successfully process notifications, including those that have already been processed.

Observed Behavior

Steps to Reproduce

I'm not sure. Even after deleting all pre-existing RawNotification records, I'm encountering this issue, but it does not appear that I actually have two duplicate emails (same exact datestamp, subject, and provider) in my inbox, so I don't understand how we're even triggering this condition.

I do have multiple emails from the same provider with the same subject line, but they have different Date: stamps on them.

Be able to defined the number of days to look back on the first execution of the Job

Proposed Functionality

By default, first time you run the Notifications Job, it doesn't limit the SINCE, so it would retrieve all the emails related to the FROM option.

Use Case

Be able to customize the number of days it looks back the first time, to not overwhelm the first run.

Deleting a Provider should not delete associated RawNotifications and ParsedNotifications

Environment

• Nautobot version: 1.1.4
• nautobot-circuit-maintenance version: 0.3.2

Expected Behavior

On deleting a Provider, circuit-maintenance records should not be auto-deleted, although any references to that Provider should be nulled out.

Observed Behavior

Deleting a Provider deletes all of its associated RawNotifications and ParsedNotifications, although CircuitMaintenances remain present since they are linked to Circuits rather than Providers.

Steps to Reproduce

This happened in a user environment where, due to provider mergers and name changes, we had multiple Provider records that actually referred to the same company. After migrating all relevant Circuits to the correct single, consolidated, Provider record, I deleted the old Provider which no longer had any associated Circuits. To my surprise, this also deleted a slew of older RawNotifications and ParsedNotifications as they were associated to the old Provider still. The result is that I still have valid CircuitMaintenance records (since the Circuits were not deleted and are now associated to the new provider) but no longer any references to the parsed data that originated these records.

I would have preferred for the old RawNotification and ParsedNotification records to remain available, presumably with null provider references, rather than being deleted.

GMail integration: optionally tag/categorize emails when successfully parsed.

Environment

• Nautobot version:
• nautobot-circuit-maintenance version:

Proposed Functionality

In the GMail integration, it would be very useful to tag processed emails with one or more categories after processing them, e.g. "parsed" and "parse_failed" perhaps.

Use Case

This makes it easier for the developer or user to filter the service inbox to see which messages were processed, skipped, and/or failed parsing for troubleshooting and data accuracy checking.

Missing "Change Log" tab from Note detail view

Environment

• Python version:
• Nautobot version:
• nautobot-circuit-maintenance version: 0.2.5

Expected Behavior

Since Notes are change-logged records, the detail view for a given Note should provide a tab linking to said changelog.

I guess it's also missing from the NotificationSource and ParsedNotification/RawNotification views, but since those are not user-editable models, I think it's less important for those (arguably they don't need change-logging at all).

Observed Behavior

No tab in the Note detail view.

Steps to Reproduce

Rendering of `raw` field in a RawNotification detail view could render text

Environment

• Nautobot version: 1.0.3
• nautobot-circuit-maintenance version: 0.1.7

Proposed Functionality

In the common case where a RawNotification's raw data is actually a human-readable string, it would be nice to render it as such, instead of as a raw byte-string. I.e., it currently appears like:

but it would be nicer to render the text, for example as:

Use Case

GMail API integration needs to support pagination

Environment

• Nautobot version: 1.1.4
• nautobot-circuit-maintenance version: 0.2.5

Proposed Functionality

The GMail API integration needs to support pagination to handle the case where a large number of emails (more than the 100 returned by default) need to be processed.

Use Case

In this particular case, I discovered this issue because the job was failing for multiple days due to #140; eventually the message that was causing the job to fail fell out of the first 100 results and so the job started passing again because the bad message was no longer being returned in the first 100 results. Unfortunately this means that numerous other older emails also never got processed.

More generally, any email inbox with a large number of providers sending notifications could easily exceed 100 new messages arriving in a given 24-hour period.

GMail integration processes emails in reverse (newest-first) order)

Environment

• Python version:
• Nautobot version:
• nautobot-circuit-maintenance version:

Expected Behavior

Emails to be processed in chronological order

Observed Behavior

Emails are processed in reverse chronological order - newest first, oldest last. This has the consequence that the terminal state of any CircuitMaintenance that has multiple emails will be that of the oldest update, instead of the newest update as it should be.

Steps to Reproduce

Change `NotificationSource` config between restarts cause DB errors

Environment

• Python version: 3.6
• Nautobot version: 1.1.2
• nautobot-circuit-maintenance version: 0.1.10

Expected Behavior

When changing the NotificationSource config between Nautobot restarts nothing wrong should happen, just update as defined.

Observed Behavior

When changing the configr from PLUGINS_CONFIG DB errors are raised:

nautobot_1       | nautobot_circuit_maintenance.models.DoesNotExist: NotificationSource matching query does not exist.
nautobot_1       | 
nautobot_1       | During handling of the above exception, another exception occurred:
nautobot_1       | 
nautobot_1       | Traceback (most recent call last):
nautobot_1       |   File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
nautobot_1       |     return self.cursor.execute(sql, params)
nautobot_1       | psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "nautobot_circuit_maintenance_notificationsource_name_key"
nautobot_1       | DETAIL:  Key (name)=(my imap source) already exists.

Steps to Reproduce

Fix yaml files to pass the yamllint step.

The following is the list of errors, which need to be fixed to pass the yamllint step:

INVOKE_NAUTOBOT_CIRCUIT_MAINTENANCE_LOCAL=True invoke yamllint
./nautobot_circuit_maintenance/fixtures/source_gmail_api.yaml
  2:10      error    string value is not quoted with double quotes  (quoted-strings)
  3:7       error    string value is not quoted with double quotes  (quoted-strings)
  8:36      error    string value is not quoted with double quotes  (quoted-strings)
  9:11      error    string value is not quoted with double quotes  (quoted-strings)
  10:11     error    string value is not quoted with double quotes  (quoted-strings)
  17:10     error    string value is not quoted with double quotes  (quoted-strings)
  18:7      error    string value is not quoted with double quotes  (quoted-strings)
  23:11     error    string value is not quoted with double quotes  (quoted-strings)
  24:11     error    string value is not quoted with double quotes  (quoted-strings)
  26:10     error    string value is not quoted with double quotes  (quoted-strings)
  27:7      error    string value is not quoted with double quotes  (quoted-strings)
  34:15     error    string value is not quoted with double quotes  (quoted-strings)
  35:11     error    string value is not quoted with double quotes  (quoted-strings)
  41:10     error    string value is not quoted with double quotes  (quoted-strings)
  42:7      error    string value is not quoted with double quotes  (quoted-strings)
  49:15     error    string value is not quoted with double quotes  (quoted-strings)
  50:11     error    string value is not quoted with double quotes  (quoted-strings)
  56:10     error    string value is not quoted with double quotes  (quoted-strings)
  57:7      error    string value is not quoted with double quotes  (quoted-strings)
  65:9      error    string value is not quoted with double quotes  (quoted-strings)
  66:10     error    string value is not quoted with double quotes  (quoted-strings)
  67:7      error    string value is not quoted with double quotes  (quoted-strings)
  75:9      error    string value is not quoted with double quotes  (quoted-strings)

./nautobot_circuit_maintenance/fixtures/handle_notifications_job.yaml
  2:10      error    string value is not quoted with double quotes  (quoted-strings)
  3:7       error    string value is not quoted with double quotes  (quoted-strings)
  8:36      error    string value is not quoted with double quotes  (quoted-strings)
  9:11      error    string value is not quoted with double quotes  (quoted-strings)
  10:11     error    string value is not quoted with double quotes  (quoted-strings)
  17:10     error    string value is not quoted with double quotes  (quoted-strings)
  18:7      error    string value is not quoted with double quotes  (quoted-strings)
  23:36     error    string value is not quoted with double quotes  (quoted-strings)
  24:11     error    string value is not quoted with double quotes  (quoted-strings)
  25:11     error    string value is not quoted with double quotes  (quoted-strings)
  32:10     error    string value is not quoted with double quotes  (quoted-strings)
  33:7      error    string value is not quoted with double quotes  (quoted-strings)
  38:36     error    string value is not quoted with double quotes  (quoted-strings)
  39:11     error    string value is not quoted with double quotes  (quoted-strings)
  40:11     error    string value is not quoted with double quotes  (quoted-strings)
  47:10     error    string value is not quoted with double quotes  (quoted-strings)
  48:7      error    string value is not quoted with double quotes  (quoted-strings)
  53:11     error    string value is not quoted with double quotes  (quoted-strings)
  54:11     error    string value is not quoted with double quotes  (quoted-strings)
  56:10     error    string value is not quoted with double quotes  (quoted-strings)
  57:7      error    string value is not quoted with double quotes  (quoted-strings)
  64:15     error    string value is not quoted with double quotes  (quoted-strings)
  65:11     error    string value is not quoted with double quotes  (quoted-strings)
  71:10     error    string value is not quoted with double quotes  (quoted-strings)
  72:7      error    string value is not quoted with double quotes  (quoted-strings)
  79:15     error    string value is not quoted with double quotes  (quoted-strings)
  80:11     error    string value is not quoted with double quotes  (quoted-strings)
  86:10     error    string value is not quoted with double quotes  (quoted-strings)
  87:7      error    string value is not quoted with double quotes  (quoted-strings)
  95:9      error    string value is not quoted with double quotes  (quoted-strings)
  96:10     error    string value is not quoted with double quotes  (quoted-strings)
  97:7      error    string value is not quoted with double quotes  (quoted-strings)
  105:9     error    string value is not quoted with double quotes  (quoted-strings)

./nautobot_circuit_maintenance/fixtures/source_imap.yaml
  2:10      error    string value is not quoted with double quotes  (quoted-strings)
  3:7       error    string value is not quoted with double quotes  (quoted-strings)
  8:36      error    string value is not quoted with double quotes  (quoted-strings)
  9:11      error    string value is not quoted with double quotes  (quoted-strings)
  10:11     error    string value is not quoted with double quotes  (quoted-strings)
  17:10     error    string value is not quoted with double quotes  (quoted-strings)
  18:7      error    string value is not quoted with double quotes  (quoted-strings)
  23:11     error    string value is not quoted with double quotes  (quoted-strings)
  24:11     error    string value is not quoted with double quotes  (quoted-strings)
  26:10     error    string value is not quoted with double quotes  (quoted-strings)
  27:7      error    string value is not quoted with double quotes  (quoted-strings)
  34:15     error    string value is not quoted with double quotes  (quoted-strings)
  35:11     error    string value is not quoted with double quotes  (quoted-strings)
  41:10     error    string value is not quoted with double quotes  (quoted-strings)
  42:7      error    string value is not quoted with double quotes  (quoted-strings)
  49:15     error    string value is not quoted with double quotes  (quoted-strings)
  50:11     error    string value is not quoted with double quotes  (quoted-strings)
  56:10     error    string value is not quoted with double quotes  (quoted-strings)
  57:7      error    string value is not quoted with double quotes  (quoted-strings)
  65:9      error    string value is not quoted with double quotes  (quoted-strings)

./development/docker-compose.requirements.yml
  11:5      warning  comment not indented like content  (comments-indentation)
  17:14     warning  too few spaces before comment  (comments)
  22:5      warning  comment not indented like content  (comments-indentation)

./development/docker-compose.base.yml
  29:17     error    string value is not quoted with double quotes  (quoted-strings)
  30:16     error    string value is not quoted with double quotes  (quoted-strings)
  31:21     error    string value is not quoted with double quotes  (quoted-strings)
  36:5      error    duplication of key "healthcheck" in mapping  (key-duplicates)
  41:17     error    string value is not quoted with double quotes  (quoted-strings)
  42:16     error    string value is not quoted with double quotes  (quoted-strings)
  43:21     error    string value is not quoted with double quotes  (quoted-strings)
  47:9      error    string value is not quoted with double quotes  (quoted-strings)
  48:9      error    string value is not quoted with double quotes  (quoted-strings)
  51:14     warning  too few spaces before comment  (comments)
  52:68     warning  too few spaces before comment  (comments)

./.github/workflows/ci.yml
  3:1       warning  truthy value should be one of [false, true]  (truth)

RawNotifications associated to wrong CircuitMaintenance

Environment

• Python version:
• Nautobot version: 1.1.3
• nautobot-circuit-maintenance version: 0.2.4

Expected Behavior

RawNotifications to be associated to the correct CircuitMaintenance records

Observed Behavior

RawNotifications are associated to the wrong CircuitMaintenance records. It looks like something isn't getting carried over or initialized properly as we see a series of consecutive RawNotifications (for different maintenance IDs) being associated to the same CircuitMaintenance instead of different ones as expected.

Steps to Reproduce

1. Not sure yet - investigating.

Add alerting features

Proposed Functionality

Provide alerting panel to enable rules to act when:

• there is a future overlapping of circuit maintenances in the same site or device that would isolate it
• ...

Provide IM integration

Use Case

Relationships are not visible in the Circuit Maintenance View

Environment

• Python version: 3.6
• Nautobot version: 1.1.0
• nautobot-circuit-maintenance version: latest

Expected Behavior

All Relationships associated with the CircuitMaintenance object should be visible in the CircuitMaintenance detail view.

Observed Behavior

User defined Relationships are not visible in the detail view page but they are visible in the edit view

Steps to Reproduce

1. Create a new relationship and associate it with the CircuitMaintenance model
2. Create a CircuitMaintenance
3. check the detail view of the newly created circuit maintenance

Use `statuses` to represent Circuit Maintenance status

Proposed Functionality

Instead of using simple text from Choices to show the status of a Circuit Maintenance leverage on custom statuses

Use Case

This would improve user experience while keeping same text functionalities.

IntegrityError if maintenance references the same circuit more than once

Environment

• Python version:
• Nautobot version: 1.1.4
• nautobot-circuit-maintenance version: 0.2.5

Expected Behavior

Job to run to completion successfully

Observed Behavior

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot_circuit_maintenance/handle_notifications/handler.py", line 166, in create_or_update_circuit_maintenance
    circuit_maintenance_entry = CircuitMaintenance.objects.get(name=maintenance_id)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 431, in get
    self.model._meta.object_name
nautobot_circuit_maintenance.models.CircuitMaintenance.DoesNotExist: CircuitMaintenance matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/newrelic/hooks/database_psycopg2.py", line 65, in execute
    **kwargs)
  File "/usr/local/lib/python3.7/site-packages/newrelic/hooks/database_dbapi2.py", line 38, in execute
    *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django_prometheus/db/common.py", line 71, in execute
    return super().execute(*args, **kwargs)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "nautobot_circuit_mainten_maintenance_id_circuit_i_01065132_uniq"
DETAIL:  Key (maintenance_id, circuit_id)=(408ed18b-9110-4c6c-9ac9-71b10555a06d, d87635fa-fca9-5130-9578-acbb0dc1bb74) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/nautobot_circuit_maintenance/handle_notifications/handler.py", line 280, in process_raw_notification
    logger, parser_maintenance, raw_entry, provider
  File "/usr/local/lib/python3.7/site-packages/nautobot_circuit_maintenance/handle_notifications/handler.py", line 187, in create_or_update_circuit_maintenance
    circuit_maintenance_entry = create_circuit_maintenance(logger, maintenance_id, parser_maintenance, provider)
  File "/usr/local/lib/python3.7/site-packages/nautobot_circuit_maintenance/handle_notifications/handler.py", line 48, in create_circuit_maintenance
    impact=circuit.impact,
  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 447, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 754, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 792, in save_base
    force_update, using, update_fields,
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 895, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 935, in _do_insert
    using=using, raw=raw,
  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1254, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1397, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/cacheops/transaction.py", line 93, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/newrelic/hooks/database_psycopg2.py", line 65, in execute
    **kwargs)
  File "/usr/local/lib/python3.7/site-packages/newrelic/hooks/database_dbapi2.py", line 38, in execute
    *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django_prometheus/db/common.py", line 71, in execute
    return super().execute(*args, **kwargs)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "nautobot_circuit_mainten_maintenance_id_circuit_i_01065132_uniq"
DETAIL:  Key (maintenance_id, circuit_id)=(408ed18b-9110-4c6c-9ac9-71b10555a06d, d87635fa-fca9-5130-9578-acbb0dc1bb74) already exists.

Steps to Reproduce

Similar to #140, except that in this case an existent circuit is referenced multiple times in the same notification email:

Presumably the fix will be similar - change CircuitImpact.objects.create() to CircuitImpact.objects.get_or_create() as appropriate.

Show `stamp` info for `CircuitMaintenances`

Environment

• Nautobot version:
• nautobot-circuit-maintenance version:

Proposed Functionality

Like the end_date or start_date the stamp_date could give information about the reception date of the notification.

Use Case

Support Gmail API Notification Source

Proposed Functionality

Add Gmail API as a supported Notification Source, supporting OAuth authentication.

See: https://www.geeksforgeeks.org/how-to-read-emails-from-gmail-using-gmail-api-in-python/

Because of the OAuth nature, the "Application", the Plugin, will require:

• the first time we set it up a manual login verification, later we could keep refreshing the token. But, we should offer a way to the user to "activate" this source. Maybe, a "TEST/VALIDATE" option could serve this goal and also provide an easy way to validate the credentials for other methods, such as IMAP.
• the way that we identify to the API is more complex than just an account id (i.e. email) and password, so it contains a "client_id", and "project_id", and a "client_secret". This means that the model should be modified to support these more complex cases.
• it should work across service restarts

Use Case

Offer an alternative to IMAP Notification Source

Support omitting the email for a provider if the provider info is contained in the notification

Environment

• Nautobot version: 1.0.0

Proposed Functionality

Make the usage of "Emails for Circuit Maintenance plugin" Custom Field for Provider optional

Use Case

So far, when using CM you need to define a custom email to filter with the FROM and classify them to each provider, assuming that this info is not always properly available from the notification and to narrow done the search.

Even this is still an interesting behaviour in terms of optimization and identification, it could be more relaxed so it could be an optional argument that if not present would behave as:

• ALL the notifications from the source will be taken into account (assuming the impact on the increase of notifications to process)
• Because the notification before being parsed has not any provider identification, when calling the circuit-maintenance-parser it could use the generic method and try to take the info from the notification if possible, or maybe add some extra metadata, such as the source email address (and match against provider well know addresses) or API urls.
• In the case that a NotificationSource it's only matching one provider, maybe we could assume that all the notifications are coming from this provider (thinking on a future API integration with an API endpoint), even this could be problematic with other types of integrations based on email boxes.

Feature: If circuit ID is an IP, do a lookup

Environment

• Nautobot version:
• nautobot-circuit-maintenance version:

Googele Fiber uses circuit IP addresses to identify their circuits rather than the ID.

Proposed Functionality

After a conversation with Christian, he has suggested we add a feature that if a circuit ID is an IP address, do a lookup in Nautobot and retrieve the ID. This logic should only be in the plugin and not the parser.

Use Case

Providers that use a circuits IP as a ID.

Summarize `ParsedNotifications` showed in the `CircuitMaintenance` view when are duplicated

Environment

• Nautobot version: 1.1.2
• nautobot-circuit-maintenance version: 0.2.4

Proposed Functionality

By default, all the ParsedNotifications related to a CircuitMaintenance are listed in the detailed view. But sometimes, notifications are kind of duplicated, just changing few words, but not changing the parsed information, so we should summarize all the ParsedNotificaitons that contain the same data in a continuous sequence.

Use Case

Clean up the relevant information related to a CircuitMaintenance

Enable configuration option to not mark emails as SEEN

Environment

• Nautobot version: 1.1.2
• nautobot-circuit-maintenance version: 0.1.9

Proposed Functionality

Offer an optional flag in the plugin config in order to change the default behaviour of change the state of an email notification retrieved to SEEN.

Use Case

In some use case, it would be desirable that the automation processes do not change the SEEN status of an email.

RawNotification detail view is not rendering properly with a large payload

Environment

• Python version: 3.6
• Nautobot version: 1.1.0
• nautobot-circuit-maintenance version: latest

Expected Behavior

RawNotification detail view should render properly for all size of RawNotification payload

Observed Behavior

django.db.utils.IntegrityError when adding multiple notes to the same maintenance with the same title

Environment

• Python version:
• Nautobot version: 1.1.4
• nautobot-circuit-maintenance version: 0.2.5

Expected Behavior

Circuit maintenance job to run to completion

Observed Behavior

django.db.utils.IntegrityError: duplicate key value violates unique constraint "nautobot_circuit_mainten_maintenance_id_title_f955987f_uniq"
DETAIL:  Key (maintenance_id, title)=(646c824b-d576-4e4c-ba19-19eb307e8d1c, Nonexistent circuit ID _) already exists.

Steps to Reproduce

1. Receive and process an email notification that references the same nonexistent circuit ID more than once - in this case this appears to be due to the parser not recognizing that "Alternate Circuit ID" needs to be used instead of "Circuit ID", and so mistakenly thinking that the circuit ID _ is being referenced twice:
   
2. Job errors out.

Either we should remove the uniqueness constraint that requires that each Note on a CircuitMaintenance must have a unique title, or else we need to explicitly detect and handle the case where a duplicate note would be created and avoid this exception.

Validate MYSQL support in tests

Environment

• Nautobot version: 1.1.4
• nautobot-circuit-maintenance version: 2.0.4

Expected Behavior

The plugin should work well with MYSQL DB

Observed Behavior

Tests are not in place so it's not clear if it works as expected.

Similar implementation: nautobot/nautobot-app-device-lifecycle-mgmt#33

Steps to Reproduce

Non ASCII Email Decoding Error

Environment

• Python version: 3.8
• Nautobot version: 1.1.3
• nautobot-circuit-maintenance version: 0.3.2

Expected Behavior

Extract email from mailbox with attachments (attachment is encoded in UTF-16).

Observed Behavior

Notification retrieval failed with unhandled Exception.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nautobot_circuit_maintenance/handle_notifications/sources.py", line 610, in get_notifications
    raw_notifications = source.receive_notifications(job_logger, since_date)
  File "/usr/local/lib/python3.8/site-packages/nautobot_circuit_maintenance/handle_notifications/sources.py", line 389, in receive_notifications
    raw_notification = self.fetch_email(job_logger, msg_id)
  File "/usr/local/lib/python3.8/site-packages/nautobot_circuit_maintenance/handle_notifications/sources.py", line 339, in fetch_email
    return self.process_email(job_logger, email_message)
  File "/usr/local/lib/python3.8/site-packages/nautobot_circuit_maintenance/handle_notifications/sources.py", line 288, in process_email
    raw_payload=email_message.as_bytes(),
  File "/usr/local/lib/python3.8/email/message.py", line 178, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/local/lib/python3.8/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/local/lib/python3.8/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/local/lib/python3.8/email/generator.py", line 218, in _dispatch
    meth(msg)
  File "/usr/local/lib/python3.8/email/generator.py", line 276, in _handle_multipart
    g.flatten(part, unixfrom=False, linesep=self._NL)
  File "/usr/local/lib/python3.8/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/local/lib/python3.8/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/local/lib/python3.8/email/generator.py", line 218, in _dispatch
    meth(msg)
  File "/usr/local/lib/python3.8/email/generator.py", line 436, in _handle_text
    super(BytesGenerator,self)._handle_text(msg)
  File "/usr/local/lib/python3.8/email/generator.py", line 253, in _handle_text
    self._write_lines(payload)
  File "/usr/local/lib/python3.8/email/generator.py", line 155, in _write_lines
    self.write(line)
  File "/usr/local/lib/python3.8/email/generator.py", line 410, in write
    self._fp.write(s.encode('ascii', 'surrogateescape'))
UnicodeEncodeError: 'ascii' codec can't encode character '\u2019' in position 916: ordinal not in range(128)

Steps to Reproduce

1. Execute job to retrieve emails from mail source.

Add support for X-Connect in addition to Circuit

Environment

• Nautobot version: 1.1.4
• nautobot-circuit-maintenance version: 0.3.2

Proposed Functionality

Add support for X-Connect in addition to Circuit. Currently X-Connect ID can be stored under a CircuitTermination.
At a minimum we should be able to match a maintenance notification to a X-Connect ID in addition to a Circuit ID.

Not sure if more changes are required

Use Case

Some X-Connect can be affected by maintenance as well, like the Metro-Connect from Equinix.

The `_raw_md5` field

What purpose does the RawNotification._raw_md5 field serve? I don't see where it's actually being used for anything, and its unique=True constraint means that trying to create RawNotifications with the same raw body text (e.g from duplicate emails) results in an exception:

duplicate key value violates unique constraint "nautobot_circuit_mainten__raw_md5_4f063303_uniq" DETAIL: Key (_raw_md5)=(d41d8cd98f00b204e9800998ecf8427e) already exists.

Use graphql extra_feature for Models

Environment

• Nautobot version:
• nautobot-circuit-maintenance version:

Proposed Functionality

Update graph_ql extra_feature for models after nautobot/nautobot#399 is fixed

Use Case

Custom Fields are not visible in Circuit Maintenance View

Environment

• Python version: 3.6
• Nautobot version: 1.1.0
• nautobot-circuit-maintenance version: latest

Expected Behavior

All custom fields associated with the CircuitMaintenance object should be visible in the CircuitMaintenance detail view.

Observed Behavior

Custom fields are not visible in the detail view page but they are visible in the edit view

Steps to Reproduce

1. Create a new custom field and associate it with the CircuitMaintenance model
2. Create a CircuitMaintenance
3. check the detail view of the newly created circuit maintenance

Migrate to Celery from RQ

Environment

• Nautobot version: 1.1.x

Expected Behavior

Use Celery instead of RQ: https://nautobot.readthedocs.io/en/latest/installation/services/#migrating-to-celery-from-rq

RawNotification is too much email dependent

Environment

• Python version: 3.8.2
• Nautobot version: 1.0.2
• nautobot-circuit-maintenance version: 0.1.3

Expected Behavior

RawNotification arguments should be more generic and not so related to email sources.

For instance:

• Subject: it's coming directly from email, it should be able to come from other sources.
• Sender: it's used today as the email sender, some NotifiicationSources won't have so it should be optional
• Source: it relates to the type of Notification Source, but maybe this could be a link to the actual NotificationSource that was used

Observed Behavior

Steps to Reproduce

Increase default `rawnotification` `raw` size to capture at lease email headers

Environment

• Python version: 3.x
• Nautobot version: 1.1.2
• nautobot-circuit-maintenance version: 0.2.4

Expected Behavior

We should be able to get the basic info from an original email when stored in the rawnotification.raw field. This data is truncated by default to 500 characters

Observed Behavior

This truncation means missing basic email headers information, so we should look for a value (8192?) that consistently give enough information in the headers to locate the notification in the email source box.

Steps to Reproduce

Add email filtering capability when using `source_header`

Proposed Functionality

When using source_header different from From it is not possible to filter emails, so you get emails that maybe are completely unrelated to the plugin, adding some noise.
Maybe, adding an extra configuration option, such as source_header_from_emails that can contain a list of the "senders", this could limit all these issues that are unrelated.

Use Case

Limit the noise from emails that are not related to the plugin work.

CircuitMaintenance "name" should probably include provider as well as maintenance_id

Environment

• Nautobot version:
• nautobot-circuit-maintenance version:

Proposed Functionality

As and when this plugin is used with an ever-growing set of providers sending maintenance notifications, it becomes increasingly likely that two different providers will send maintenance notifications that happen to use the same maintenance ID number/string. (Consider the case of two providers each sending their very first maintenance notification, and both choosing to identify it as maintenance id "00000001"). Because we use the maintenance_id as the globally-unique name of the CircuitMaintenance model instances, these two disparate sets of maintenance notifications would inadvertently be merged into a single CircuitMaintenance instance named 000000001, which is almost certainly undesirable.

To avoid collision between providers in this fashion, we should probably include the notification provider value in addition to the maintenance_id in constructing the CircuitMaintenance name.

Use Case

Circuit Impact view does not render objects correctly

Environment

• Python version: 3.8.5
• Nautobot version: 1.1.3
• nautobot-circuit-maintenance version: 0.2.1

Expected Behavior

Circuit ID and object link is displayed along with the Circuit Impact. Maintenance should also be linked in the displayed name.

Observed Behavior

Steps to Reproduce

1. Execute circuit maintenance job to retrieve emails.
2. In the job output, for a successful circuit maintenance, click on the Circuit Impact link for the circuit /circuit-maintenance/impact/<id>/.

Unable to navigate directly to the circuit maintenance from the Circuit page

Environment

• Python version: 3.6
• Nautobot version: 1.1.0
• nautobot-circuit-maintenance version: latest

Expected Behavior

From the detail view of a circuit i should be able to visualize the upcoming maintenances and directly navigate to them by following a link.

Observed Behavior

The list of upcoming maintenances is represented in the detail view of a circuit on the left but there is no link to the corresponding object so it's not possible to navigate to it

"since" filter is too coarse-grained

Environment

• Python version:
• Nautobot version:
• nautobot-circuit-maintenance version:

Expected Behavior

Running the job should pick up all emails that arrived since the last time the job was run

Observed Behavior

Because the since filtering is date-based (not date-time based), any emails that arrived during the same day as the last job run will be skipped over, even though they haven't been processed yet.

Steps to Reproduce

The `get_since_reference` provides is not consistent enough

Environment

• Python version:
• Nautobot version: 1.1.3
• nautobot-circuit-maintenance version: 0.2.5

Expected Behavior

The reference time to get notifications should be the last one that we got raw_notifications. Because now the date is matching the STAMP or the DATE of the email, this causes that a wrong STAMP, maybe in the future, will avoid retrieving notifications

Observed Behavior

Steps to Reproduce

1. Fake a ICS notificaitons with a STAMP in the future
2. Run the job and the since will be in the future

More flexible mapping of Provider (Nautobot) to Provider (circuit-maintenance-parser)

Environment

• Nautobot version:
• nautobot-circuit-maintenance version:

Proposed Functionality

Mapping Nautobot Providers to circuit-maintenance-provider Provider classes based on the Nautobot Provider slug is inflexible as it relies on enforcing that the Nautobot Provider slug be exactly equal to the circuit-maintenance-provider Provider class __name__. In practice this may not always be the case, and it's not always going to be feasible to change the existing slug in Nautobot to resolve this.

Use Case

I have a Nautobot deployment with circuits defined for provider CenturyLink (former name of the provider now called Lumen). Without changing the slug of this provider, or creating a new Lumen provider and remapping all of the circuits to this new provider, I have no way to make emails from Lumen be handled by the Lumen parser provider because the plugin is looking for a parser called centurylink.

integrate with capacity metrics plugin to expose the maintenance status of a circuit as a metric

Proposed Functionality

It would be very useful to integrate with the capacity metrics plugin to expose the maintenance status of a circuit as a metric.

The goal is to expose a metric that will reflect the maintenance status of each circuit, something similar to Admin Status and Oper status for interfaces

• 1 circuit is operational
• 2 circuit is in maintenance mode

# Circuit operational
circuit_maintenance_status{"circuit": "XXXXX", provider="ntt", circuit_type="peering", site='XX"} 1.0

# Circuit in maintenance mode
circuit_maintenance_status{"circuit": "YYYYYY", provider="ntt", circuit_type="peering", site='YY"} 2.0

This is something we are already doing with (few) other plugins, here is an example for the chatops plugin
https://github.com/nautobot/nautobot-plugin-chatops/blob/develop/nautobot_chatops/__init__.py#L59

Use Case

I can think of 2 main use cases for this feature:

• Visualize in Grafana the maintenance status of a circuit overtime.
• Identify when a circuit is in maintenance to avoid generating an alert, from the alertmanager for example.

Custom Fields are not visible in the Raw Notification View

Environment

• Python version: 3.6
• Nautobot version: 1.1.0
• nautobot-circuit-maintenance version: latest

Expected Behavior

All custom fields associated with the RawNotification object should be visible in the RawNotification detail view.

Observed Behavior

Custom fields are not visible in the detail view page

Steps to Reproduce

1. Create a new custom field and associate it with the RawNotification model
2. Create a RawNotification (via the ORM)
3. check the detail view of the newly created raw notification

Add Plugin metrics about instance performance

Environment

• Nautobot version: 1.1.4
• nautobot-circuit-maintenance version: 0.3.2

Proposed Functionality

Leverage on prometheus_client to expose key performance metrics for the plugin.

Use Case

Track the performance of the plugin.

Add reporting

Proposed Functionality

Provide a Reporting feature in order to get data about:

• Number of maintenances
• Hours of downtime

per circuit/per provider

This data could be available via UI, API, metrics or PDF report

Use Case

Add all Providers to NotificationSources which class parent is EmailSource

Proposed Functionality

by default, all Providers will be added as related Providers for NotificationSources which parent class is EmailSource, as it's the usual use-case, where all the notifications are received in one email box.
Obviously, if the Provider don't have an email configured, it will be skipped from the loop.

Use Case

This simplifies the usability of the plugin with this assumption.

Emit Django Signals when a Circuit Maintenance is going to "start" and when it "ends"

Environment

• Nautobot version: 1.1.0

Proposed Functionality

This proposal is to emit a Django Signal from Circuit Maintenance plugin when one of the Circuit Maintenances scheduled, in a valid state (CONFIRMED) is going to START (with a configurable time in advance in case some automation should have to interact with the network) and when the event passed the END time.

This new feature could leverage on the new Celery task queue, that every X time is checking for the state of the Circuit Maintenances and sends the proper signals.

Use Case

With this feature, other Django components could be passively listening to Circuit Maintenance events and perform actions, such as changing Circuit state or sending IM messages automatically.

Add test coverage to the Custom Field to map a Provider to a different provider in the parser library

There should be a test coverage for handling notifications where there is a Custom mapping to the parser library, and verify that the whole processing is correct.

In CircuitMaintenance detail view, Notification "Date" should not be created/updated timestamp

Environment

• Python version:
• Nautobot version:
• nautobot-circuit-maintenance version: 0.2.4

Expected Behavior

In the CircuitMaintenance detail view, raw notifications associated with a CircuitMaintenance should be listed based on their notification timestamp.

Observed Behavior

Raw notifications are listed based on their created/updated timestamp, not the notification timestamp.

Note that all four notifications show times of 9/20 at 2:48 or 2:49 PM as that's the time that the parser job was run and processed these notifications. Clicking through to each individual notification is necessary in order to show its true notification timestamp:

Steps to Reproduce

1. Run circuit maintenances job with data available in GMail
2. View details of any given CircuitMaintenance record.

Source Object Validation Invalid Results if Session Exists

Environment

• Python version: 3.8
• Nautobot version: 1.0.3
• nautobot-circuit-maintenance version: 0.1.7

When validating a notification source using the test_authentication method on the CLI, if a session already exists the return method for the test will always be True - Login OK

Expected Behavior

Ignore session data when testing the notification source. It may be required to change attributes in the source during initial setup and source validation.

Observed Behavior

Running test_authentication() will always return True after initial attempt as the session data causes a conditional check in the logic to avoid subsequent connection attempts.

Steps to Reproduce

1. Open CLI session and import nautobot
2. from nautobot_circuit_maintenance.handle_notifications.sources import Source
3. Create a test source and execute test_authentication

from nautobot_circuit_maintenance.handle_notifications.sources import Source
>>> source = Source.init(name="TEST IMAP")
>>> source.test_authentication()
(False, 'LOGIN failed.')
>>> source.test_authentication()
(True, 'Test OK')

Leverage on Nautobot Scheduler to run it periodically

Proposed Functionality

When Nautobot implements a scheduler (nautobot/nautobot#275), Circuit Maintenance should enable a scheduling to retrieve notifications automatically.

Use Case

This will enable a continuous operation of the plugin without operation interaction.

Improve Documentation

Environment

• Python version: 3.6
• Nautobot version: 1.0.1
• nautobot-circuit-maintenance version: 0.1.1

In the add notifications form I am prompted for a Url with no description and source type is either gmail or unknown. A minimal description of the Url field and some documentation around configuring other email sources would be very beneficial.

Expected Behavior

The documentation should provide examples for configuring a POP3 or an IMAP mail server at a minimum.

Observed Behavior

Documentation refers to URL but nothing further