diff --git a/plugins/events-backend/src/service/hub/createEventBusRouter.ts b/plugins/events-backend/src/service/hub/createEventBusRouter.ts index d9c2bdf3dc..0ebd25d304 100644 --- a/plugins/events-backend/src/service/hub/createEventBusRouter.ts +++ b/plugins/events-backend/src/service/hub/createEventBusRouter.ts @@ -30,6 +30,93 @@ import { EventParams } from '@backstage/plugin-events-node'; const DEFAULT_NOTIFY_TIMEOUT_MS = 55_000; // Just below 60s, which is a common HTTP timeout +/* + +# Event Bus + +This comment describes the event bus that is implemented here in the events +backend, and by default used by the events service. + +## Overview + +The events bus implements a subscription mechanism where subscribers must exist +upfront for events to be stored. It uses a single inbox for all events, with +each subscriber having its own pointer for how far into the inbox it has read. + +In order to avoid busy polling, the API uses a long-polling mechanism where a +request is left hanging until the client should try to read again. + +The event bus is not implemented with any guarantees of events being consumed, +but it does aim to make it unlikely that events are dropped + +## API + +### POST /bus/v1/events + +This endpoint is used to publish new events to the event bus on a specific +topic. It can optionally include a set of subscription IDs for subscribers that +have already been notified of the event. This is to enable an optimization where +we notify subscribers locally if possible, and avoid the need for events to be +relayed through the events bus at all of possible. + +For an event to be published and stored there must already exist a subscriber +that is subscribed to the event's topic, and that hasn't already been notified +of the event. If no such subscriber is found, the event will be discarded. + +### PUT /bus/v1/subscriptions/:subscriptionId + +This endpoint is used to create or update a subscriptions. Subscriptions are +shared across the entire bus and divided by subscription ID. Multiple clients +can be reading events from the same subscription at the same time, but only one +of those clients will receive each event. This enables division of work by using +the same subscriber ID across multiple instances, as well as broadcasting by +ensuring separate subscribers IDs. + +### GET /bus/v1/subscriptions/:subscriptionId/events + +This endpoint is used to read events from a subscription. It will return a batch +of events for the subscribed topics that have not yet been read by the +subscription. If no such events are available, the endpoint will return a 202 +response and then hang end response until an event is available or a timeout is +reached. This allows clients to call this endpoint in a loop but will keep +traffic overhead fairly low. + +## Delivery guarantees + +When reading events from the event bus, clients should always implement a +graceful shutdown where they process any events that are received from the +events endpoint before shutting down. This is also the reason that the events +endpoint does not return any events when responding with a 202 blocking the +response, because there would otherwise be a race condition where the events +might be lost in transit if the client shuts down. By always sending an empty +response and requiring the client to send another request, we ensure that the +client is prepared to halt shutdown until the request had been fully processed. + +## Local processing optimization + +When possible, events will be processed locally before sent to the event bus. +The client will also inform the bus of which subscriptions have already been +notified of the event, so that the bus can completely avoid storing an event if +it has already been fully consumed by all subscribers. + +## Automated cleanup & event window + +Events are deleted once they are outside the guaranteed storage window. By +default the window 10 minutes for all events, and 24 hours for the last 10000 +events. This ensures that the event log doesn't grow indefinitely, while still +allowing subscribers with restarts or outages to catch up to past events, +ensuring that events are likely not lost. + +Subscriptions are also cleaned up if their read pointer falls outside of the +current event window. This ensures that stale subscribers don't accumulate and +cause unnecessary storage of events. + +*/ + +/** + * Creates a new event bus router + * @internal + */ export async function createEventBusRouter(options: { logger: LoggerService; database: DatabaseService;