openedx / openedx-events Goto Github PK
View Code? Open in Web Editor NEWOpen edX events from the Hooks Extensions Framework
License: Apache License 2.0
Open edX events from the Hooks Extensions Framework
License: Apache License 2.0
This is a placeholder issue to start discussion on this topic. There is a desire to create a single “tracking log” OpenEdxPublicSignal that can be used for all tracking log events to facilitate event handling and transformation. Specifically openedx/wg-data#28 calls for this, but several other xAPI use cases are in various levels of ideation.
The general idea is that once #210 is done we would like to add this new signal, and introduce a few other changes as outlined in the epic above to support configurable an event logs -> xAPI pipeline (as well as other consumers) that operators can tailor to their needs. Given the volume of tracking logs we're hoping that we can filter logs by event name on the producer side as part of this work.
Currently, arch-bom, a team at edX, is testing MVP Kafka event bus integration into the Open edX platform. During the implementation, a couple of questions were raised and github issues seems like a great public venue to have the discussion.
Questions:
My main goal for this discussion is pinpointing what to use to create a schema for Kafka. So far, I've been using the Attrs decorated classes in data.py.
CMS_COURSE_PUBLISHED
(name open to discussion) OpenEdxPublicSignal in openedx-events with the fields listed below (with appropriate data types)Fields:
blocks_url
effort
end
enrollment_start
enrollment_end
id
media: { banner_image: {uri, uri_absolute}, course_image: {uri}, course_video: {uri}, image: {raw, small, large} }
name
number
org
short_description
start
start_display
start_type
pacing
mobile_available
hidden
invitation_only
course_id
Note: CMS_COURSE_PUBLISHED is just a suggestion for a name and can be improved on. We should make sure we are consistent with "CMS" vs "Studio" nomenclature.
Open Questions:
Where in the openedx-events module structure should this new event live? @ormsbee does tCRIL have a strong opinion on this?
Break up the functionality in the AvroAttrsBridge to make responsibilities more clear
Acceptance Criteria:
Original (private) Jira ticket: https://2u-internal.atlassian.net/browse/ARCHBOM-2102
At this time, producing a signal to the event bus requires code. See the event bus how-to for details.
This is problematic because:
This issue proposes to make the switch to configuring this via settings, rather than code.
Some thoughts/questions about the settings:
I'm not clear if this is really an issue for the implementation repos (Kafka or Redis), or for openedx-event where the consumer API interface is defined. However, at some point a breaking change was made to this interface, and we did not have proper test coverage to prevent us from breaking (at least) one of the implementations.
This ticket is to understand what went wrong and how this could be prevented in the future. This may also include doc enhancements regarding the right way to create and test an implementation.
In openedx/edx-platform#32249, we had a discussion about the notifications
app that veered into message bus topic/event mapping questions that deserved its own separate conversation.
@robrap wrote:
Thanks @ormsbee. Without my getting lost in notifications design considerations, using existing events, new events, and the event bus all makes sense to me.
I've certainly read that only one service should be responsible for writing to any given topic. I'm not sure if the following question applies to this use case, but is this a useful rule or not? If it comes up, we'd need to determine if it makes sense to have a shared topic that any service could write to and is used as a single pipe into some other service, or if we have a service that subscribes to all of the topics that include some event, like USER_NOTIFICATION, and either responds immediately or writes it to a new consolidated topic. This is just an implementation detail about how to use the event bus, and isn't about whether or not to use the event bus.
@ormsbee replied:
I've certainly read that only one service should be responsible for writing to any given topic.
I think it's sane as long as it's either one writer with potentially many consumers, or many writers with one consumer. For auditing purposes, things like PII would also make sense to have many writers, one consumer.
If it comes up, we'd need to determine if it makes sense to have a shared topic that any service could write to and is used as a single pipe into some other service, or if we have a service that subscribes to all of the topics that include some event, like USER_NOTIFICATION, and either responds immediately or writes it to a new consolidated topic.
I would favor having a single topic for all user notifications to go to, partitioned by a hash of the user.
Apologies, but I haven't been following event type -> topic mapping discussions. I had thought that it would be based on the event types themselves, and not the sending or receiving apps. I don't want to derail this ticket into that discussion, but do you have a link to the ticket where that was hashed out? I vaguely recall skimming something a while back, but I can't seem to find it.
@robrap replied:
- When producing, you produce an event (with an event type) to a topic. Unfortunately, it is possible to produce an event to the wrong topic, but no one is likely to listen to it, so it will just die out once fixed.
- When consuming, the management command consumes from a topic. All event types on the topic will be converted to signals, and only those signals with receiver code will be handled. See https://openedx.atlassian.net/wiki/spaces/AC/pages/3508699151/How+to+start+using+the+Event+Bus#Consuming-the-signal.
I wouldn't be able to point to specific issues around this except the recent #78, which was around how to implement the capability of sending multiple event types to a single topic.I think it's sane as long as it's either one writer with potentially many consumers, or many writers with one consumer.
I imagine that the many writers to one consumer pattern would have a definite initial consumer and purpose in mind. However, I'm not sure that a second consumer might not come along and say "Hey, that's the perfect set of events that I want for my own purpose." Would it be an issue for there to be a second consumer?
If we want to allow many producers producing to a single topic, I think the important point would be that both producers and consumers of that topic should have this shared understanding from the start, and it should be well documented. In other words, it would not be ok for a service to see an event that it could also fire, and just start firing it into a previously existing topic.
This epic is part of the Event Bus work documented in openedx/platform-roadmap#28.
This epic can be used for future event bus tasks that don't yet fall into an existing planned epic/milestone. Tasks can be moved from this milestone to a planned milestone as-needed.
Milestone Details:
[Event Bus] Future
See https://github.com/openedx/openedx-events/issues/74 for Epic details.
Open issues/PRs with this milestone: https://github.com/pulls?q=is%3Aopen+milestone%3A%22%5BEvent+Bus%5D+Future%22
As per https://github.com/openedx/openedx-events/blob/da9a2a9df07b49bc7760970f58e76de86611ee82/docs/decisions/0009-course-catalog-info-changed-design.rst , we are only using the course catalog info changed signal for setups that include publisher, and we are not including any fields that are not present in studio when publisher is enabled. effort
is one of these fields, so we should remove it from the event data.
Hello!
This repo doesn't have a codecov setup for the Open edX organization, what can we do to configure it?
Acceptance Criteria:
workers
vs webapp
if challenging)Notes:
OEP-41 Asynchronous Server Event Message Format also dictates the use of the CloudEvents specification. Combined with this ADR, we would be required to adhere to the CloudEvents Avro Format. There may also be additional CloudEvent related work tied to a particular protocol binding, like the Kafka Protocol Binding for CloudEvents. This, however, is out of scope of this particular decision.
Our current approach has been to generally follow the CloudEvents spec as we understand it where and when we need to make a choice. However, we are not putting attention to ensuring that we have a complete or purely correct implementation. It is unclear how to do so, and what the benefits would be of such an effort at this time.
For current references, see:
The purpose of this issue is to communicate the current approach. If someone wants to put in more effort to review/implement any gaps around this, that help is welcome.
FYI: @ormsbee: Since you were the author of the OEP. Note: if you have no issues with this, we can close this issue.
Add a unit test that loops over all events and generates the Avro schema. Note: if any events have an issue, we may need a way to label them as "not for use with the event bus", and have the test skip them.
A/C:
The repo readme must comply with the OEP-55 specifications for phase 1 maintenance program.
Hello!
I have a non-open source project within the edX ecosystem that I would like to use kafka/the event bus with; however, I would ideally not need to have to define events in an open source repo for a private project (not that it's secret per se, but just that it would be mysterious).
Curious what kinds of thoughts you have on the matter!
Thanks,
Chris
This field only makes sense in setups where publisher is disabled. As noted in https://github.com/openedx/openedx-events/blob/main/docs/decisions/0009-course-catalog-info-changed-design.rst , we are specifically designing the event for setups that include publisher.
At the moment, event if short_description information is sent, it will be ignored by the course discovery consumer.
send_event_with_custom_metadata
currently takes metadata keys individually as kwargs, but it turns out to be awkward, as in practice in event-bus-kafka we end up building an EventsMetadata
and then splatting it into the call along with the data dict: https://github.com/openedx/event-bus-kafka/blob/51b0055990ed0116dc06781d6e1c8f90959eb088/edx_event_bus_kafka/internal/consumer.py#L365
This also increases risk of collisions. #177 would address that, but alternatively we could just accept a single metadata
arg (positionally?) as a breaking change.
For a new adopter, it should be clear:
A/C
We reached a working consensus that we would depend on all implementations from any IDA that needed to use the Event Bus, but never made an ADR out of it: #88 (comment)
Let's finish that process.
Currently, each IDA that wants to publish events must add a @receiver
-decorated function that listens for a signal and calls the event-bus producer (specifying a topic and key-field as well). It would probably be better to instead have openedx-events do this automatically. Perhaps we could have settings like the following:
EVENT_BUS_AUTOSEND = [
{
'signal': 'org.openedx.content_authoring.course.catalog_info.changed.v1',
'topic': 'course-catalog-info-changed',
'key_field': 'catalog_info.course_key',
},
...
]
This is a continuation of #77 (see discussion). As part of that ticket, it was decided to defer minorversion (extension), as detailed in OEP-41.
The minorversion field has been noted in [Evolution] Discovery: Schema Evolution edx/edx-arch-experiments#53, but this ticket was also created in case that detail gets missed.
In the ADR (https://github.com/openedx/openedx-events/blob/main/docs/decisions/0004-external-event-bus-and-django-signal-events.rst), we discussed firing the same event signal after consuming an event bus event.
An advantage of this proposal is it mimics what is done for in-process events.
However, there is a major difference between in-process events and events in the consuming application. For in-process events, we must use a technology like celery to get asynchronous processing of the events. However, our event bus consuming applications are already asynchronous in relation to the main user/api-facing service.
For these applications, we have an opportunity to simply do the event processing in-process from the point of view of the consuming application, and to horizontally scale the consumer containers based on processing requirements. We could do this without spinning up separate celery tasks that will then compete with other tasks in the same celery queue.
Note, I personally would like to take advantage of processing in the consumers and avoiding celery to start. We don't need to make any final decisions here, but it would be nice to have a nice initial recommendation.
Thank you.
When using a data class with an optional datetime field, the AvroSignalSerializer will error if the field value is not present or is None (even though it's supposed to be optional).
The following test can be used to recreate the issue:
class AttrsWithDefaultDate:
birthday = attr.ib(type=datetime, default=None)
def test_serialization_of_optional_date_field(self):
SIGNAL = create_simple_signal({"test_data": AttrsWithDefaultDate})
serializer = AvroSignalSerializer(SIGNAL)
test_data = {"test_data": AttrsWithDefaultDate()}
serialize_event_data_to_bytes(test_data, SIGNAL)
self.assertNull(test_data["test_data"]["birthday"])
In the current event bus code, it is possible for signals to be broadcast as events using varying key IDs on the same topic. For example, the CERTIFICATE_CREATED signal could be emitted as events with a key taken from certificate.user.id
, or a key taken from certificate.course.course_key
. This presents a problem for deserialization: One of those is a number, and the other is a string; deserialization of the key will fail if the key schema is not knowable from the event.
I can see several possible solutions:
"key_field": "certificate.user.id"
) so that the consumer can tell how to deserialize the key.
Trying to add python3.10 support on eox-hooks fails due to openedx-events.
ERROR: test_receiver_called_after_cert_change (eox_hooks.tests.test_receivers.TestCohortEventsHandler)
Test that hooks_handler is called the correct information after sending
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/django/test/utils.py", line 387, in inner
return func(*args, **kwargs)
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/mock/mock.py", line 1346, in patched
return func(*newargs, **newkeywargs)
File "/home/edunextdev/Documents/edunext/eox-hooks/eox_hooks/tests/test_receivers.py", line 481, in test_receiver_called_after_cert_change
COHORT_MEMBERSHIP_CHANGED.send_event(
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/openedx_events/tooling.py", line 170, in send_event
format_responses(responses, depth=2),
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/openedx_events/utils.py", line 90, in format_responses
).pformat(obj)
File "/usr/lib/python3.10/pprint.py", line 157, in pformat
self._format(object, sio, 0, 0, {}, 0)
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/openedx_events/utils.py", line 31, in _format
if isinstance(obj, collections.Callable):
AttributeError: module 'collections' has no attribute 'Callable'
======================================================================
ERROR: test_receiver_called_after_enroll_change (eox_hooks.tests.test_receivers.TestEnrollmentEventsHandler)
Test that hooks_handler is called the correct information after sending
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/django/test/utils.py", line 387, in inner
return func(*args, **kwargs)
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/mock/mock.py", line 1346, in patched
return func(*newargs, **newkeywargs)
File "/home/edunextdev/Documents/edunext/eox-hooks/eox_hooks/tests/test_receivers.py", line 263, in test_receiver_called_after_enroll_change
COURSE_ENROLLMENT_CHANGED.send_event(
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/openedx_events/tooling.py", line 170, in send_event
format_responses(responses, depth=2),
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/openedx_events/utils.py", line 90, in format_responses
).pformat(obj)
File "/usr/lib/python3.10/pprint.py", line 157, in pformat
self._format(object, sio, 0, 0, {}, 0)
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/openedx_events/utils.py", line 31, in _format
if isinstance(obj, collections.Callable):
AttributeError: module 'collections' has no attribute 'Callable'
======================================================================
ERROR: test_receiver_called_after_enrollment (eox_hooks.tests.test_receivers.TestEnrollmentEventsHandler)
Test that hooks_handler is called the correct information after sending
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/django/test/utils.py", line 387, in inner
return func(*args, **kwargs)
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/mock/mock.py", line 1346, in patched
return func(*newargs, **newkeywargs)
File "/home/edunextdev/Documents/edunext/eox-hooks/eox_hooks/tests/test_receivers.py", line 231, in test_receiver_called_after_enrollment
COURSE_ENROLLMENT_CREATED.send_event(
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/openedx_events/tooling.py", line 170, in send_event
format_responses(responses, depth=2),
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/openedx_events/utils.py", line 90, in format_responses
).pformat(obj)
File "/usr/lib/python3.10/pprint.py", line 157, in pformat
self._format(object, sio, 0, 0, {}, 0)
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/openedx_events/utils.py", line 31, in _format
if isinstance(obj, collections.Callable):
AttributeError: module 'collections' has no attribute 'Callable'
======================================================================
ERROR: test_receiver_called_after_unenrollment (eox_hooks.tests.test_receivers.TestEnrollmentEventsHandler)
Test that hooks_handler is called the correct information after sending
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/django/test/utils.py", line 387, in inner
return func(*args, **kwargs)
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/mock/mock.py", line 1346, in patched
return func(*newargs, **newkeywargs)
File "/home/edunextdev/Documents/edunext/eox-hooks/eox_hooks/tests/test_receivers.py", line 295, in test_receiver_called_after_unenrollment
COURSE_UNENROLLMENT_COMPLETED.send_event(
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/openedx_events/tooling.py", line 170, in send_event
format_responses(responses, depth=2),
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/openedx_events/utils.py", line 90, in format_responses
).pformat(obj)
File "/usr/lib/python3.10/pprint.py", line 157, in pformat
self._format(object, sio, 0, 0, {}, 0)
File "/home/edunextdev/Documents/edunext/eox-hooks/env/lib/python3.10/site-packages/openedx_events/utils.py", line 31, in _format
if isinstance(obj, collections.Callable):
AttributeError: module 'collections' has no attribute 'Callable'
This is a bit of a discovery ticket.
Acceptance Criteria:
Notes:
Questions:
event_type
, event definition (signal), data class, etc. for the proposed solution?Bots that help with maintenance work must be correctly configured and working. Bots like:
Some of the events specified in openedx-events contain PII. One clear example is the UserPersonalData class.
AC:
The repo catalog-info must show correct/valid information about the project's current status, and the maintainers can find information on:
- Seeing what you own.
- Seeing what other people own.
- Seeing how repos relate to each other.
We should consider adding a how-to for adding events, that may pull from and reference ADRs, but would be a clear how-to document.
One potentially missing detail, is whether all Data objects should have the suffix "Data"? See
for one example that doesn't match this convention. Separate and related, do we want linting to help enforce any conventions?Do we want to document known subdomains? (I have additional questions about this, but better left to a separate issue.)
We need a way to load event-bus consumer implementations based on configuration, so that IDAs aren't hardcoded to use one particular implementation. (See #87 for the producer side.)
There are a couple of ways we might go about this:
INSTALLED_APPS
.
This issue serves as a starting point for the discussion mentioned during the Open edX Events guide pull request, saying organization proprietary events might live in this repo as well.
Original comment: #80 (comment)
The design intent behind eduNEXT/openedx-events was to be able to reuse the event definitions for both internal events (signals) and external events (message bus).
Currently, the plan seems to have all events be signals. Some of these signals would be received by a Kafka producer, which will then publish the data in the signal to the event bus. The Kafka consumer would then listen to the event and convert it back to a signal.
This would mean:
The definitions for the django signals are defined in signal.py. Specifically, django signals definition in encapsulated in OpenEdxPublicSignal class. OpenEdxPublicSignal is a django signal classs.
How sure are we that we want a signal event for every Kafka event?
Does OpenEdxPublicSignal actually need to be a signal? Even if we decide to start with Event Bus events always being sent downstream of the signal (for now), does it make sense to enforce this in the class design and leave no room for adjustments?
This class overrides some signal methods to get users to use a special method.
a. Is it possible for this class to just contain a signal class instead (when needed), or can the signal class not be dynamic?
Could OpenEdxPublicSignal be responsible for sending both signal and Kafka event bus events?
This complexity may not be worth it now without a strong need, but it could be noted as a possibility. For now, maybe this should be documented as a rejected alternative, simply because it isn’t yet required.
Could we want the design of an event to look different when internal vs external? For example, how will we design these lifecycle events? See ARCHBOM-2008
Signals implemented and sent before timestamp was an available parameter are possibly not meeting the time requirements as documented in OEP-41.
This issue proposes a one-time audit and cataloging of which implementations should be updated. That work could potentially be separately ticketed. I know things will be missed with the current process, where I try to remember to ask others when they put events on the event bus. The XBLOCK_X related events are soon to be produced to the event bus, and I'm not clear if there is an appropriate timestamp to be used.
FYI: @ormsbee
This epic is part of the Event Bus work documented in openedx/platform-roadmap#28.
The epic covers remaining work for having the new use case in Production, after completing happy-path and sad-path work in other epics. The use case implemented is for course publishing from Studio that could be consumed by Discovery, potentially replacing a small piece of the Refresh Course Metadata batch job. For the purposes of this Epic, we will handle any remaining work, including:
Also, adding some preparation work for others coming on to the Event Bus.
Milestone Details:
[Event Bus] Production ready event
See https://github.com/openedx/openedx-events/issues/91 for Epic details.
Open issues/PRs with this milestone: https://github.com/pulls?q=is%3Aopen+milestone%3A%22%5BEvent+Bus%5D+Production+ready+event%22
Right now, the EventsMetadata object is hardcoded to identify the source as "openedx/{service}/web." In order to be compliant with https://open-edx-proposals.readthedocs.io/en/latest/architectural-decisions/oep-0041-arch-async-server-event-messaging.html#id3, we should be distinguishing between web and workers. It's not immediately obvious to me where we can get this information. The only setting I can find that distinguishes web from worker is NEW_RELIC_APP_NAME, which seems very fragile.
This epic is part of the Event Bus work documented in openedx/platform-roadmap#28.
The epic covers the implementation of the sad-paths for a new use case. The current plan is to implement the use case of course publishing from Studio that could be consumed by Discovery, potentially replacing a small piece of the Refresh Course Metadata batch job. For the purposes of this Epic, we will handle any error or exceptional conditions. This may include:
The code developed during this epic will have an eye toward reusability and pluggability, but will err toward getting something worked so that the API can be defined in a later iteration.
Milestone Details:
[Event Bus] Implement sad-path
See https://github.com/openedx/openedx-events/issues/90 for Epic details.
Open issues/PRs with this milestone: https://github.com/pulls?q=is%3Aopen+milestone%3A%22%5BEvent+Bus%5D+Implement+sad-path%22
The signal parameter of the consumer init method and the management command have been deprecated and should be removed. It may be necessary to make them optional first, release that, then remove them as a new major version.
A/C:
We ran into an issue where fastavro was updated in services and started error-ing on events: #137. It turns out these errors were for a new validation check in fastavro that caught something we were doing that it did not expect. Our unit tests did begin failing with this change.
One idea is to constrain fastavro in openedx-events so that services would not have upgraded until tests passed in this repo. Here is a sample PR for that: #139.
The upside to constraining would be that a fastavro bug would not be introduced that fails our tests.
The downside is that fastavro would require manual upgrades, and we don't have a great process for ensuring that would happen. Additionally, in the above case, in theory we would have been producing/consuming invalid data and would have had a challenging time upgrading. But, maybe that would have been better than finding out through a set of failed event consumption.
The event_metadata parameter was left as optional in the send() method to avoid a complicated roll-out. Once edx-platform has been updated to use the new parameter we should go back and make it mandatory.
This epic is part of the Event Bus work documented in openedx/platform-roadmap#28.
The epic covers the implementation of the happy-path for a new use case. The current plan is to implement the use case of course publishing from Studio that could be consumed by Discovery, potentially replacing a small piece of the Refresh Course Metadata batch job. For the purposes of this Epic, it may be enough to just consume and log the events in Discovery.
The epic will include any work to make the existing event bus boiler plate code more reusable across repos, rather than having it duplicated from the existing edx-platform experiments plugin and the existing license-manager code.
Milestone Details:
[Event Bus] Implement happy-path
See https://github.com/openedx/openedx-events/issues/63 for Epic details.
Open issues/PRs with this milestone: https://github.com/pulls?q=is%3Aopen+milestone%3A%22%5BEvent+Bus%5D+Implement+happy-path%22
The edx_event_bus_kafka
dependency will be specified and pinned in a file outside of each IDA that uses it, since openedx-events will load the implementation by looking up a module reference in the IDA's config. This means that dependency version changes on the library will not be versioned with the IDA (i.e. in the requirements/base.txt files) and we'll need some sort of expand-contract approach to API changes.
For example, we might need to do something like this:
KafkaProducer
class in event-bus-kafka and delegate the existing send_to_event_bus
to use itsend_to_event_bus
function from event-bus-kafkaEvery master-branch deployer would need to do this in lockstep. (Maybe that's just 2U?) For Open edX releases we might be able to get away with just saying "use the release tag".
send_robust
catches exceptions raised by receivers and returns them (which are then passed through send_event
as well), but this means that errors don't make it into New Relic (or other telemetry). This is a general problem with Django, but it may be particularly troublesome in the context of signal teleportation on the event bus, as the receiver may begin failing (perhaps due to a new bug or a change in the data) and the errors would not raise alerts or be available for debugging. Being able to detect a broken producer/consumer pair and fix it will be important.
The solution here may be to set a custom attribute, call NR's notice_exception
, or even ask NR to instrument send_robust
(although that would only solve the issue for NR users).
Someone (not me, but I am completely blanking on who at the moment 😞 ) brought up the idea of marking new events "experimental" for a certain period of time after initial introduction so that we have a little more flexibility to change things before we mark them as "supported" for long term usage. I thought this was a good idea, but I'm curious what others thought.
FYI: @mariajgrimaldi, @felipemontoya, @feanil, @robrap
Ticket to keep track of the work done around the maintainers pilot.
There's a need to:
We need a way to load event-bus implementations based on configuration, so that IDAs don't have to depend directly on those libraries.
This issue is to define and implement an interface that allows an IDA to retrieve a producer instance by looking at Django settings. Consumer side has been split off to #147 since it is lower priority.
edx_event_bus_kafka
module that are present: openedx/edx-platform#31356Check over the PR for the unit test on signal events being serialized/deserialized to generate the Avro schema
It is possible (though moderately unlikely) that an IDA could inadvertently be configured to both produce a signal to the event bus and also consume that signal from the event bus. This would result in any such event being reproduced and circulated indefinitely on a tight loop, causing quadratically increasing load and eventually an outage.
There are a couple of ways we could avoid this:
Any such safeguard would not be able to detect anything that explicitly receives, recreates, and re-sends a signal, but that sort of thing is probably not worth worrying about.
We recently ran into an issue where we want our event bus signal consumer logic to know the broker's timestamp of the message. However, that is not available in the signal handler, and brings up a number of issues:
time
metadata with now()
as the implementation, which would give the consumer time.A/C:
Additional tasks:
send_event_with_metadata
.send_event_with_metadata
for consumers.A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.