openwisp / openwisp-notifications Goto Github PK
View Code? Open in Web Editor NEWNotifications module of OpenWISP
Home Page: http://openwisp.org
License: GNU General Public License v3.0
Notifications module of OpenWISP
Home Page: http://openwisp.org
License: GNU General Public License v3.0
This happened while testing on openwisp-monitoring.
[2020-07-31 03:58:02,003: ERROR/ForkPoolWorker-5] Task openwisp_notifications.tasks.delete_obsolete_notifications[6a347c48-b253-4f77-bf73-430a48810e0a] raised unexpected: DoesNotExist('ContentType matching query does not exist.')
Traceback (most recent call last):
File "/home/nemesis/.virtualenvs/openwisp2/lib/python3.7/site-packages/django/contrib/contenttypes/models.py", line 19, in get_by_natural_key
ct = self._cache[self.db][(app_label, model)]
KeyError: 'default'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/nemesis/.virtualenvs/openwisp2/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/nemesis/.virtualenvs/openwisp2/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
return self.run(*args, **kwargs)
File "/home/nemesis/Code/openwisp/openwisp-notifications/openwisp_notifications/tasks.py", line 15, in delete_obsolete_notifications
instance_app_label, instance_model
File "/home/nemesis/.virtualenvs/openwisp2/lib/python3.7/site-packages/django/contrib/contenttypes/models.py", line 21, in get_by_natural_key
ct = self.get(app_label=app_label, model=model)
File "/home/nemesis/.virtualenvs/openwisp2/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/nemesis/.virtualenvs/openwisp2/lib/python3.7/site-packages/django/db/models/query.py", line 417, in get
self.model._meta.object_name
django.contrib.contenttypes.models.ContentType.DoesNotExist: ContentType matching query does not exist.
Ensure the exception is catched and return from the task silently in these cases (it will happen when content types are removed from the DB).
Follow up to #44 (comment)
Two issues:
static
like the other lines, why? https://github.com/openwisp/openwisp-notifications/blob/dev/openwisp_notifications/templates/admin/base_site.html#L29-L30I tried the following in base_site.html
which seems to solve the issue, please double check and improve the solution if needed:
{% extends "admin/base_site.html" %}
{% load i18n static notification_tags %}
{% block extrastyle %}
{{ block.super }}
<link rel="stylesheet" type="text/css" href="{% static 'openwisp_notifications/css/loader.css' %}" />
<link rel="stylesheet" type="text/css" href="{% static 'openwisp_notifications/css/notifications.css' %}" />
{% endblock extrastyle %}
{% block extrahead %}
{{ block.super }}
{% if 'jquery.js' not in media.js %}
<script type="text/javascript" src="{% static 'admin/js/vendor/jquery/jquery.js' %}"></script>
<script type="text/javascript" src="{% static 'admin/js/jquery.init.js' %}"></script>
{% endif %}
{% endblock %}
{% block userlinks %}
<div class="ow-notifications" id="openwisp_notifications">
<img class="bell-icon" src="{% static 'openwisp_notifications/images/icons/icon-bell.svg' %}" />
{% unread_notifications %}
</div>
{% csrf_token %}
{% notification_widget %}
{% endblock %}
{% block footer %}
{{ block.super }}
<script type="text/javascript">
{% if OPENWISP_NOTIFICATIONS_HOST %}
const notificationApiHost = new URL('{{ OPENWISP_NOTIFICATIONS_HOST }}');
{% else %}
const notificationApiHost = window.location;
{% endif %}
const notificationSound = new Audio('{{ OPENWISP_NOTIFICATIONS_SOUND }}');
</script>
<script type="text/javascript" src="{% static 'openwisp_notifications/js/vendor/reconnecting-websocket.min.js' %}"></script>
<script type="text/javascript" src="{% static 'openwisp_notifications/js/notifications.js' %}"></script>
{% endblock footer %}
Please add a testcase for this bug which fails without the fix
Happens randomly in tests (it can be seen sometimes on travis as well).
Instead of using a function, notify_handler
can be implemented as a class which will provide ability to customise it without being the need to rewrite it completely. We can break it into small static methods to perform different aspects of generating a notification, like figuring out recipients, creating notification objects, etc. __call__
method can be used to make the class callable. This will make it extendable/customizable.
By default ajax requests will be sent to /
, we need to make sure this is configurable, so we may be able to point out another domain.
Notifications should be received and displayed on admin dashboard without reloading the page. A message(toast) should also be to notify upon receiving a new notification.
Currently, we are relying upon openwisp-monitoring
for manually testing new features which are added to this module. We can ease this part by adding a model to the sample app.
Idea
Add a dummy model with name and organisation fields in the sample app. Register a notification type for this model.
A notification should be generated when objects of this model are created or deleted using this notification type. Notification should be received only by users of the same organisation.
Thanks to @NoumbissiValere for bringing this up, ๐
Switch test runner to openwisp_utils.tests.TimeLoggingTestRunner.
Current implementation:
When a user opens any admin webpage. an API call to server is made through AJAX and notifications are fetched.
When user scrolls through the notification widget, more notifications are fetched from the server.
The notifications are stored on runtime memory of the browser to prevent subsequent calls to server.
Notification is marked read by sending a message through web sockets.
Shortcomings
Proposed Solution
@pandafy have you enabled the debug toolbar?
See here: http://openwisp.io/docs/developer/hacking-openwisp-python-django.html#django-debug-toolbar
Have you noticed the amount of queries?
Before we were storing the message but now we're generating it each time which generates far too many queries, so we should find a solution to this using caching (and we should handle cache invalidation).
one possible way of doing is the following :
actor
object and try to retrieve it from the cache before queryingtarget
Once the problem is fixed, we should add assertNumQueries in tests to make sure the amount of queries is kept under control.
I will arrange this in the board so it has less priority over the API and the JS widget.
Goals:
NotificationType
to categorize notifications.Project Idea
Add the concept of notification type, it may be a list with some default values which can then be extended by each module.
Each notification type shall have a message which is generated from a configurable text template, the text shall be markdown formatted, the markdown text shall be expanded into HTML when the notification is sent via email or viewed via the JS widget.
Discussion
Project idea says to have some kind of list, in my opinion having a model will be more appropriate for this use case. .
This is appearing in a VM running openwisp-notifications dev branch.
raised unexpected: RuntimeError('You cannot use AsyncToSync in the same thread as an async event loop - just await the async function directly.')
Traceback (most recent call last):
File "/opt/openwisp2/env/lib/python3.8/site-packages/celery/app/trace.py", line 412, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/openwisp2/env/lib/python3.8/site-packages/celery/app/trace.py", line 704, in __protected_call__
return self.run(*args, **kwargs)
File "/opt/openwisp2/env/lib/python3.8/site-packages/openwisp_notifications/tasks.py", line 27, in delete_obsolete_notifications
Notification.objects.filter(where).delete()
File "/opt/openwisp2/env/lib/python3.8/site-packages/django/db/models/query.py", line 722, in delete
deleted, _rows_count = collector.delete()
File "/opt/openwisp2/env/lib/python3.8/site-packages/django/db/models/deletion.py", line 337, in delete
signals.post_delete.send(
File "/opt/openwisp2/env/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 173, in send
return [
File "/opt/openwisp2/env/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 174, in <listcomp>
(receiver, receiver(signal=self, sender=sender, **named))
File "/opt/openwisp2/env/lib/python3.8/site-packages/openwisp_notifications/handlers.py", line 172, in clear_notification_cache
ws_handlers.notification_update_handler(
File "/opt/openwisp2/env/lib/python3.8/site-packages/openwisp_notifications/websockets/handlers.py", line 13, in notification_update_handler
async_to_sync(channel_layer.group_send)(
File "/opt/openwisp2/env/lib/python3.8/site-packages/asgiref/sync.py", line 76, in __call__
raise RuntimeError(
RuntimeError: You cannot use AsyncToSync in the same thread as an async event loop - just await the async function directly.
In #46 , dummy model was added to test whether notifications are being created and deleted.
Add documentation for this.
PUT /api/v1/notification/ignore/<content-type>/<object-id>/
)DELETE /api/v1/notification/ignore/<content-type>/<object-id>/
)Project Idea
Allow users to disable notifications for specific objects, either permanently or until a specific date, with predefined set of durations: permanently, 1 day, 3 days, 1 week, 1 month, custom date. In the case the disabling is temporary, the setting should be deleted automatically after expiration by a simple celery task which is run periodically with celery beat.
Add feature list and screenshots.
This should be done in the last phase of the project, or it can be done gradually (each time you add a feature you list it and document it, the screenshots can be added as last).
Goals
Project Idea
Allow users to disable notifications by notification types, right now it's only possible to enable or disable the notifications globally.
For each notification type and organization the user should be able to disable either all notifications or only the email notification.
Example: Receive notifications for organization "default"?
Two checkboxes: Web and Email; if email is checked also web is checked automatically (because the notification depends on the basic notification being created).
Such a solution would give us the possibility to implement mobile push notifications in the future by adding a new checkbox for mobile notifications.
Allow users to disable notifications for specific objects, either permanently or until a specific date, with predefined set of durations: permanently, 1 day, 3 days, 1 week, 1 month, custom date. In the case the disabling is temporary, the setting should be deleted automatically after expiration by a simple celery task which is run periodically with celery beat.
@pandafy always use reverse
when generating URLs, I see there are some occurrences of hardcoded urls in test_api.py
, if there are other occurrences fix those as well please.
Currently, register_notification_type functions requires type_name
to be provided separately.
Add type_name
key in notification type configuration and use it instead of requiring an argument while registering notification type.
Goals
Project Idea
Create a javscript widget for the admin which allows to view all notification and expand their details by clicking on it.
This widget shall support infinite scrolling (we have a nice implementation in django-ipam which can be reused, either partially or totally) and expand the markdown into HTML.
The operation of opening a notification detail shall flag it as read.
This feature will require adding API endpoints to get notification list and details.
Useful resources:
Edited by @pandafy
How to replicate:
Expected result:
Actual result:
reconnecting-websocket.min.js:3 WebSocket connection to 'ws://localhost:8000/ws/notifications/' failed: Error during WebSocket handshake: Unexpected response code: 403
open @ reconnecting-websocket.min.js:3
a @ reconnecting-websocket.min.js:3
(anonymous) @ notifications.js:4
jquery.js:4046 jQuery.Deferred exception: Cannot read property 'scrollHeight' of undefined TypeError: Cannot read property 'scrollHeight' of undefined
at onUpdate (http://localhost:8000/static/openwisp_notifications/js/notifications.js:124:58)
at initNotificationWidget (http://localhost:8000/static/openwisp_notifications/js/notifications.js:165:9)
at notificationWidget (http://localhost:8000/static/openwisp_notifications/js/notifications.js:177:5)
at HTMLDocument.<anonymous> (http://localhost:8000/static/openwisp_notifications/js/notifications.js:14:9)
at mightThrow (http://localhost:8000/static/admin/js/vendor/jquery/jquery.js:3762:29)
at process (http://localhost:8000/static/admin/js/vendor/jquery/jquery.js:3830:12) undefinednnection to 'ws://localhost:8000/ws/notifications/' failed: Error during WebSocket handshake: Unexpected response code: 403
open @ reconnecting-websocket.min.js:3
(anonymous) @ reconnecting-websocket.min.js:3
If no organization and no recipient is provided in notify.send()
then notifications will be sent to all users who have opted-in to receive notifications (notificationuser__receive=True
).
openwisp-notifications/openwisp_notifications/handlers.py
Lines 32 to 51 in d2b2dd8
The problem arises at this statement
where = where | (Q(is_superuser=True) & Q(notificationuser__receive=True))
which update where as
(OR: ('notificationuser__receive', True), (AND: ('is_superuser', True), ('notificationuser__receive', True)))
A straight forward solution will be something like this
where = Q()
if target_org:
where = Q(is_staff=True) & Q(openwisp_users_organization=target_org) & Q(notificationuser__receive=True)
if recipient:
# Check if recipient is User, Group or QuerySet
if isinstance(recipient, Group):
recipients = recipient.user_set.filter(notificationuser__receive=True)
elif isinstance(recipient, (QuerySet, list)):
recipients = recipient
else:
recipients = [recipient]
else:
where = where | (Q(is_superuser=True) & Q(notificationuser__receive=True))
recipients = (
User.objects.select_related('notificationuser')
.order_by('date_joined')
.filter(where)
)
While this works, I have doubts that follows best practices since notificationuser__receive=True
is redundant in this case. I would like to know your thoughts before proceeding.
This bug got introduced in #7
I was referring to #19 for openwisp-monitoring/#86. Nice work in this PR ๐
I noticed a minor typo here so opened this issue ๐
...models.Site` ---> ...models.Site``
CC: @pandafy
As the amount of notifications generated by openwisp and the related rules about who to send to will grow, this will slow down the existing operations.
Every time an operation that can potentially generate a notification is executed (eg: when the notify signal is used), the calculations have to be performed and then acted upon, which may slow down execution when a large number of users are involved.
I think we should modify the notify signal to pass all the operations to a celery worker so the notification calculation and sending can happen entirely in the background.
This should not be hard, we have all the pieces of the puzzle in place, tests will execute celery tasks with EAGER set to True so it won't make a difference.
In settings.py we'd need the following as we have in monitoring and fw-upgrader:
if not TESTING:
CELERY_BROKER_URL = 'redis://localhost/1'
else:
CELERY_TASK_ALWAYS_EAGER = True
CELERY_TASK_EAGER_PROPAGATES = True
CELERY_BROKER_URL = 'memory://'
@pandafy can you investigate this please?
Keep in mind that when invoking a celery task, we shouldn't pass objects to it but only strings, integers, floats, because the values are serialized and passed from one process to another one.
Add an animated gif to the readme which shows the notification widget in openwisp-monitoring.
This should be done right before the eng of GSoC, but wait for my greenlight to proceed please.
I have launched a firmware upgrade operation on an instance where I upgraded to the latest dev versions of openwisp-monitoring and openwisp-notifications and I started noticing notifications which we have not introduced yet, eg:
Giving up, device not reachable anymore after upgrade
timed out
I wonder WTF is happening?
Provided that this is a feature we do want to introduce, but as far as I know by the last time I reviewed the code, we were not doing this explicitly, so I think something wrong is going on here and it should be fixed.
@pandafy do you have any idea?
@NoumbissiValere @Vivekrajput20 FYI โ๏ธ
Project Idea
Prepare a general email notification template which is configurable (can be changed with a setting) and defaults to using the default logo of OpenWISP (available in openwisp_utils.admin_theme). Make the logo configurable and add a way to supply extra CSS if needed.
Presently for a derivative app, we need to define admin/base_site.html
so that it can work properly.
The main reason for this is presence of app label openwisp_notifications
in url
tag.
A way to solve this will be to override ModelAdmin
and pass app label as a variable.
Copy structure of README from another module and adapt it to this module.
We'll be adding more info to the README as we go along.
Remove the GitHub action for managing issues on Project board after GSoC 2020.
In our use case with openwisp-monitoring, the url
parameter is only passed for device
. Since we are already building target_url
for target object in openwisp-notifications, therefore url
parameter can be removed to keep things simple.
Goals
Project Idea
Add celery beat task to automatically delete notifications older than X days, X defaulting to 90, configurable via setting
Project Idea
Create a javscript widget for the admin which allows to view all notification and expand their details by clicking on it.
This widget shall support infinite scrolling (we have a nice implementation in django-ipam which can be reused, either partially or totally) and expand the markdown into HTML.
The operation of opening a notification detail shall flag it as read.
This feature will require adding API endpoints to get notification list and details.
Should be done after #3
Instead of providing link to notification, provide link to related object. On opening this link, this notification should be automatically marked as read.
Update setup instructions in README.
Explicitly mention ordering of INSTALLED_APPS.
Use override_settings
or mock.patch
to set OPENWISP_NOTIFICATION_HTML_EMAIL
to False in one test and write a test that ensures this feature works as expected.
Change static sub dir from openwisp_notifications/
to openwisp-notifications/
for consistency with other ow modules.
@pandafy I think that after the API is done (#48) we can issue the first release of the module and publish it to pypi.
This should push us to make everything prettier (eg: eliminate all "TODO" placeholders from the readme) and will help the rest of the OpenWISP community to take this effort more seriously.
Once #48 is merged, can you please prepare a summary of the most relevant features we added in the change log (see CHANGES.rst
in the other modules) please? And set the version to 0.1.0 final. I'll then proceed to publish it on pypi and add you as a maintainer (create an account there if you don't have one yet).
Thanks in advance.
After merging #29.
The notification bell icon doesn't render properly:
I suggest trying on openwisp-monitoring with the latest master of openwisp-notifications.
the widget shows always 10 notifications, even if there are more, it doesn't seem to load new data
The current way we implemented reading notifications is showing its limits: notifications are marked as read as soon as the widget is opened, which doesn't give us a chance to distinguish between read and unread ones.
A possible step forward could be to flag them as read on hover. The alternative could be leave it as it was before, so that they're marked unread only if clicked, and is up to the user to click on "mark all as read" if they don't want to click over every one.
Clicking on a notification sends me to HTTPS also in dev mode.
@pandafy what do you think?
By double checking the code here: https://github.com/openwisp/openwisp-notifications/blob/dev/openwisp_notifications/handlers.py#L52, I think there is a bug in the following scenario:
When notifications are generated for org B without specifying recipients, the user will get notifications even though they shouldn't.
The solution is to add OrganizationUser.is_admin=True
to the where clause.
This issue has lower priority.
Please pin down the requirements being used:
https://github.com/openwisp/openwisp-notifications/blob/master/requirements.txt
Also, for the django-rest-framework dependency, do it like we're doing here:
Mixed Content: The page at 'https://URL/admin/config/' was loaded over HTTPS, but attempted to connect to the insecure WebSocket endpoint 'ws://URL/ws/notifications/'. This request has been blocked; this endpoint must be available over WSS.
Do not hardcode ws://...
. The entire base url should be configurable, not just the host, and we have to make sure defaults to wss if settings.DEBUG
is False
.
The subject of the email is truncated to 24 characters.
Toasts are really great and very useful, they will allow us to do great things and inform the user in real time about what is going on in the network.
But we need to make the current implementation more robust.
I deleted some metrics, then went to the notifications page and this exception was raised:
AttributeError: 'NoneType' object has no attribute 'name'
We need to make this code more resilient to failures:
https://github.com/openwisp/openwisp-notifications/blob/master/openwisp_notifications/base/models.py#L54-L59
Any item may be missing for some reason.
Try to replicate it by generating notifications related to a metric in openwisp-monitoring and then delete the metric, try deleting any of the generic relations
A few things need to be done:
Suggestions:
Full stack trace:
KeyError: 'display_list_message'
File "django/db/models/options.py", line 581, in get_field
return self.fields_map[field_name]
FieldDoesNotExist: Notification has no field named 'display_list_message'
File "django/contrib/admin/utils.py", line 262, in lookup_field
f = _get_non_gfk_field(opts, name)
File "django/contrib/admin/utils.py", line 293, in _get_non_gfk_field
field = opts.get_field(name)
File "django/db/models/options.py", line 583, in get_field
raise FieldDoesNotExist("%s has no field named '%s'" % (self.object_name, field_name))
AttributeError: 'NoneType' object has no attribute 'name'
File "django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "django/core/handlers/base.py", line 145, in _get_response
response = self.process_exception_by_middleware(e, request)
File "django/core/handlers/base.py", line 143, in _get_response
response = response.render()
File "django/template/response.py", line 105, in render
self.content = self.rendered_content
File "django/template/response.py", line 83, in rendered_content
return template.render(context, self._request)
File "django/template/backends/django.py", line 61, in render
return self.template.render(context)
File "django/template/base.py", line 171, in render
return self._render(context)
File "django/template/base.py", line 163, in _render
return self.nodelist.render(context)
File "django/template/base.py", line 936, in render
bit = node.render_annotated(context)
File "django/template/base.py", line 903, in render_annotated
return self.render(context)
File "django/template/loader_tags.py", line 150, in render
return compiled_parent._render(context)
File "django/template/base.py", line 163, in _render
return self.nodelist.render(context)
File "django/template/base.py", line 936, in render
bit = node.render_annotated(context)
File "django/template/base.py", line 903, in render_annotated
return self.render(context)
File "django/template/loader_tags.py", line 150, in render
return compiled_parent._render(context)
File "django/template/base.py", line 163, in _render
return self.nodelist.render(context)
File "django/template/base.py", line 936, in render
bit = node.render_annotated(context)
File "django/template/base.py", line 903, in render_annotated
return self.render(context)
File "django/template/loader_tags.py", line 150, in render
return compiled_parent._render(context)
File "django/template/base.py", line 163, in _render
return self.nodelist.render(context)
File "django/template/base.py", line 936, in render
bit = node.render_annotated(context)
File "django/template/base.py", line 903, in render_annotated
return self.render(context)
File "django/template/loader_tags.py", line 62, in render
result = block.nodelist.render(context)
File "django/template/base.py", line 936, in render
bit = node.render_annotated(context)
File "django/template/base.py", line 903, in render_annotated
return self.render(context)
File "django/template/loader_tags.py", line 62, in render
result = block.nodelist.render(context)
File "django/template/base.py", line 936, in render
bit = node.render_annotated(context)
File "django/template/base.py", line 903, in render_annotated
return self.render(context)
File "django/contrib/admin/templatetags/base.py", line 33, in render
return super().render(context)
File "django/template/library.py", line 214, in render
_dict = self.func(*resolved_args, **resolved_kwargs)
File "django/contrib/admin/templatetags/admin_list.py", line 342, in result_list
'results': list(results(cl)),
File "django/contrib/admin/templatetags/admin_list.py", line 318, in results
yield ResultList(None, items_for_result(cl, res, None))
File "django/contrib/admin/templatetags/admin_list.py", line 309, in __init__
super().__init__(*items)
File "django/contrib/admin/templatetags/admin_list.py", line 232, in items_for_result
f, attr, value = lookup_field(field_name, result, cl.model_admin)
File "django/contrib/admin/utils.py", line 271, in lookup_field
value = attr(obj)
File "openwisp_notifications/admin.py", line 116, in display_list_message
return strip_tags(obj.message)
File "django/utils/functional.py", line 48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "openwisp_notifications/base/models.py", line 55, in message
md_text = config['message'].format(notification=self)
We are generating links to related objects at multiple places.
We can instead create a property for these links and use them at other places.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.