diff --git a/beps/0012-metrics-service/README.md b/beps/0012-metrics-service/README.md new file mode 100644 index 0000000000..9948f22cba --- /dev/null +++ b/beps/0012-metrics-service/README.md @@ -0,0 +1,351 @@ +--- +title: Backstage Metrics Service +status: implementable +authors: + - '@kurtaking' +owners: + - '@kurtaking' +project-areas: + - core-framework + - observability +creation-date: 2025-06-23 +--- + +## Table of Contents + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) +- [Naming Conventions](#naming-conventions) +- [Design Details](#design-details) + - [References](#references) + - [Integration with OpenTelemetry Auto-Instrumentation](#integration-with-opentelemetry-auto-instrumentation) + - [Configuration](#configuration) + - [Interface](#interface) + - [Root Metrics Service](#root-metrics-service) + - [Plugin Metrics Service](#plugin-metrics-service) + - [Example](#example) +- [Release Plan](#release-plan) +- [Dependencies](#dependencies) +- [Alternatives](#alternatives) + +## Summary + +Add a core `MetricsService` to Backstage's framework to provide a unified interface for metrics instrumentation. The service offers OpenTelemetry-based capabilities with support for app configuration (e.g. `app-config.yaml`). + +This approach leverages industry standards while focusing the `MetricsService` on distinct Backstage concerns, following the same pattern as other core services (DatabaseService builds on Knex, LoggerService builds on Winston, HttpRouterService builds on Express, etc.). + +## Motivation + +There is no guidance when it comes to metrics instrumentation. While individual plugins may implement their own metrics, there's no standardized approach for metrics collection, naming, or configuration. + +By providing a core metrics service: + +- **Plugin Authors** and the **Community** gain a straightforward way to address metrics instrumentation and can focus on business logic instead of needing to reimplement metrics plumbing. +- **Backstage Admins** receive a reliable stream of metrics from the core system for monitoring, alerting, and troubleshooting. + +### Goals + +- Plugin-scoped metric namespacing +- Consistent metrics patterns across all plugins +- Aligned with OpenTelemetry industry standards +- Provide a familiar interface as other core services +- Standardized initialization and lifecycle management + +The catalog and scaffolder plugins will be updated to use the new metrics service in the initial alpha release. + +### Non-Goals + +- Adding metrics to plugins that don't currently have it (outside of catalog and scaffolder) +- Tracing and other telemetry concerns are out of scope for this BEP. +- Refactoring the existing `LoggerService`. Future work to unify observability related concerns would be ideal, but not a goal. + +## Proposal + +Following similar patterns to other core services, create a new `RootMetricsService` responsible for initializing the OpenTelemetry SDK, root-level concerns, and the creation of plugin-specific `MetricsService` instances. The root service delegates to the plugin-scoped `MetricsService` to initialize a meter for each registered plugin based on the `service.name` and optional `service.version` provided in the app's `app-config.yaml` file. This is based on the recommendation in the OpenTelemetry [documentation](https://opentelemetry.io/docs/languages/js/instrumentation/#acquiring-a-meter). + +## Naming Conventions + +All Backstage metrics follow this hierarchical pattern: + +`backstage.{scope}.{scope_name}.{metric_name}` + +**Where:** + +- `backstage` is the root namespace for all Backstage metrics +- `{scope}` is the system scope (either **plugin** or **core**) +- `{scope_name}` is the name of the plugin or core service (e.g., `catalog`, `scaffolder`, `database`) +- `{metric_name}` is the hierarchical metric name as provided by the plugin author (e.g., `entities.processed.total`, `tasks.completed.total`) + +### Scope + +The `scope` represents where it belongs in the Backstage ecosystem. + +- `plugin` - A plugin-specific metric (e.g. `backstage.plugin.catalog.entities.count`) +- `core` - A core service metric (e.g. `backstage.core.database.connections.active`) + +### Plugin-Scoped Metrics + +Pattern: `backstage.plugin.{pluginId}.{metric_name}` + +```yaml +# Examples +backstage.plugin.catalog.entities.processed.total +backstage.plugin.scaffolder.tasks.completed.total +backstage.plugin.techdocs.builds.active +backstage.plugin.auth.sessions.active.total # todo: technically a core service and a backend plugin +``` + +### Core Metrics + +Pattern: `backstage.core.{core_service}.{metric_name}` + +```yaml +# Examples +backstage.core.database.connections.active +backstage.core.scheduler.tasks.queued.total +backstage.core.httpRouter.requests.total +``` + +## Design Details + +### References + +- [General naming considerations](https://opentelemetry.io/docs/specs/semconv/general/naming/#general-naming-considerations) +- [Acquiring a meter](https://opentelemetry.io/docs/languages/js/instrumentation/#acquiring-a-meter) + +### Integration with OpenTelemetry Auto-Instrumentation + +The `RootMetricsService` will automatically enable instrumentation for known libraries leveraged by the Backstage framework. Configuration will be provided to enable or disable auto-instrumentation via inclusion or exclusion lists. + +- Express +- Knex +- Winston +- etc. + +The `MetricsService` **complements** rather than duplicates auto-instrumentation by focusing on **application-level metrics** that only Backstage can provide. For example, the catalog plugin may want to track the number of entities processed by the `refresh` operation and the kind of entity being processed. + +```ts +// Auto-instrumentation provides (automatically): +// - http.server.requests.total{method="GET", route="/catalog/entities", status_code="200"} +// - http.server.request.duration{method="GET", route="/catalog/entities"} + +// MetricsService provides (manually): +const entityMetrics = metricsService.createCounter('entities.processed.total'); +entityMetrics.add(entities.length, { operation: 'refresh', kind: 'Component' }); +``` + +### Configuration + +```ts +// Not final, but this is the general idea... +interface MetricsConfig { + enabled: boolean; + + resource: { + serviceName?: string; + serviceVersion?: string; + environment?: string; + }; + + collection?: { + exportIntervalMillis?: number; + // ... + }; + + exporters: Array<{ + type: 'prometheus' | 'otlp' | 'console' | '...'; + config?: Record; + }>; + + autoInstrumentation: { + enabled: boolean; + include?: string[]; + exclude?: string[]; + }; +} +``` + +```yaml +backend: + metrics: + enabled: true + + resource: + serviceName: backstage + serviceVersion: 0.0.1 + environment: production + + # Collection settings + collection: + exportIntervalMillis: 15000 + + exporters: + - type: prometheus + config: + port: 9464 + # ... + - type: console + + autoInstrumentation: + enabled: true + exclude: ['express'] +``` + +### Interface + +Provide a wrapper around OpenTelemetry's API while re-exporting the types from the `@opentelemetry/api` package. This introduces concepts already familiar to both the Backstage community and those familiar with OpenTelemetry. + +```ts +interface MetricsService { + // Synchronous instrumentation + createCounter(name: string, options?: MetricOptions): Counter; + createUpDownCounter(name: string, options?: MetricOptions): UpDownCounter; + createHistogram(name: string, options?: MetricOptions): Histogram; + createGauge(name: string, options?: MetricOptions): Gauge; + + // Asynchronous instrumentation + createObservableCounter( + name: string, + options?: MetricOptions, + ): ObservableCounter; + createObservableUpDownCounter( + name: string, + options?: MetricOptions, + ): ObservableUpDownCounter; + createObservableGauge(name: string, options?: MetricOptions): ObservableGauge; + + // Future - add additional convenience methods as we learn more about the needs of the framework +} +``` + +#### Root Metrics Service + +The `RootMetricsService` is responsible for initializing the OpenTelemetry SDK and creating plugin-scoped metrics services. If the end user wants to initialize their own SDK, they are responsible for initializing the OpenTelemetry SDK with their own configuration. The `RootMetricsService` is responsible for providing metrics to other root services and creating plugin-scoped metrics services. + +```ts +interface RootMetricsService { + forPlugin(pluginId: string): MetricsService; +} + +export const rootMetricsServiceFactory = createServiceFactory({ + // depends on as little as possible so that it can be initialized as early as possible. + service: rootMetricsServiceRef, + deps: { + rootConfig: coreServices.rootConfig, + }, + factory: ({ rootConfig }) => { + return DefaultRootMetricsService.fromConfig(rootConfig); + }, +}); +``` + +```ts +class DefaultRootMetricsService implements RootMetricsService { + private sdk: NodeSDK; + + static fromConfig(config: Config): RootMetricsService { + const metricsConfig = config.getOptionalConfig('backend.metrics'); + + const sdk = new NodeSDK({ + resource: createResourceFromConfig(metricsConfig), + instrumentations: [ + getNodeAutoInstrumentations({ + ...getAutoInstrumentationConfig(metricsConfig), + }), + ], + metricReader: createMetricReadersFromConfig(metricsConfig), + }); + + sdk.start(); + + return new DefaultRootMetricsService(sdk); + } + + constructor(private sdk: NodeSDK) {} + + forPlugin(pluginId: string): MetricsService { + return new PluginMetricsService(pluginId); + } + + async shutdown(): Promise { + await this.sdk.shutdown(); + } +} +``` + +#### Plugin Metrics Service + +Each plugin receives a metrics service that automatically namespaces all metrics to match the naming conventions. + +```ts +const metricsServiceFactory = createServiceFactory({ + service: metricsServiceRef, + deps: { + rootMetrics: coreServices.rootMetrics, + pluginMetadata: coreServices.pluginMetadata, + }, + factory: ({ rootMetrics, pluginMetadata }) => { + return rootMetrics.forPlugin(pluginMetadata.getId()); + }, +}); + +class PluginMetricsService implements MetricsService { + // ... + constructor(private pluginId: string) { + this.meter = metrics.getMeter(`backstage.plugin.${pluginId}`); + } + + // ... other interface methods + + private prefixMetricName(name: string): string { + return `backstage.plugin.${this.pluginId}.${name}`; + } +} +``` + +##### Example + +```ts +const entitiesProcessed = metricsService.createCounter( + 'entities.processed.total', + { + description: 'Total entities processed during refresh', + unit: '{entity}', + }, +); + +entitiesProcessed.add(100); + +// ... +// metric is now available as `backstage.plugin.catalog.entities.processed.total` +``` + +## Release Plan + +1. Create a new `RootMetricsService` that initializes the OpenTelemetry SDK and creates plugin-scoped metrics services. +1. Create the plugin-scoped `MetricsService` that provides a metrics service for plugins. +1. Create alpha-related documentation to add to existing core service [docs](https://backstage.io/docs/backend-system/core-services/index). +1. Release the metrics service under `@alpha`. +1. Mark all existing metrics implementations as deprecated. +1. Refactor catalog and scaffolder plugins to use the new (alpha) `MetricsService`. +1. Offer a migration path for existing adopters to migrate to the new metrics service. +1. Release the metrics service +1. Update all documentation to reference the new metrics service. +1. Create follow-up action items to integrate the new metrics service into the core system. +1. Fully deprecate all existing metrics implementations like the existing Prometheus one-off implementations. + +## Deprecation Plan + +TBD + +## Dependencies + +1. The root metrics service MUST BE initialized as EARLY as possible to prevent dependents from receiving no-op meters +1. There are one-off implementations of metrics in the wild that may conflict with the proposed service. However, this is unlikely to be a problem as the SDK should continue to pick things up. + +## Alternatives + +Plugin authors continue to implement their own metrics as they see fit. diff --git a/beps/0012-telemetry-service/README.md b/beps/0012-telemetry-service/README.md deleted file mode 100644 index c316d05ca8..0000000000 --- a/beps/0012-telemetry-service/README.md +++ /dev/null @@ -1,190 +0,0 @@ ---- -title: Backstage Telemetry Service -status: implementable -authors: - - '@kurtaking' -owners: - - '@kurtaking' -project-areas: - - backend - - core - - framework -creation-date: 2025-06-23 ---- - -# BEP: Backstage Telemetry Service - -- [Summary](#summary) -- [Motivation](#motivation) - - [Goals](#goals) - - [Non-Goals](#non-goals) -- [Proposal](#proposal) -- [Design Details](#design-details) - - [Configuration](#configuration) -- [Release Plan](#release-plan) -- [Dependencies](#dependencies) -- [Alternatives](#alternatives) - -## Summary - -Add a core telemetry service to Backstage's framework that provides a unified interface for metrics, tracing, and logging instrumentation. The service offers OpenTelemetry-based capabilities with zero configuration defaults while allowing extensibility through app configuration (e.g. `app-config.yaml`) and plugin modules. - -## Considerations - -1. It's possible a single telemetry service is too encompassing and that more granular services are needed. For example, a `MetricsService` and `TracingService` which would work along side the existing `LoggerService` to solve for telemetry needs. - -## Motivation - -There is currently no guidance for plugin authors when it comes to telemetry and while individual plugins may implement their own metrics or tracing, there's no unified service that provides a standardized approach for telemetry and observability. - -By providing a core telemetry service: - -- **Plugin Authors** gain a straightforward, documented way to address telemetry, and can focus on business logic instead of needing to reimplement telemetry plumbing. -- The **community** receives a standard approach to telemetry. -- **Backstage Operators/SRE/Platform Engineers** receive a consistent, reliable stream of metrics from the core system, simplifying monitoring, alerting, and troubleshooting. They can enforce metrics standards, aggregate data, and build dashboards more easily. -**Organizations** with their own observability or compliance requirements can override the core service to integrate with proprietary or custom telemetry solutions, without forking or modifying plugins. - -### Goals - -Provide a unified service that provides a standardized approach for telemetry where setup docs like [these](https://backstage.io/docs/tutorials/setup-opentelemetry/) are not needed. - -- Consistent telemetry patterns across all plugins -- Zero-configuration defaults that work out of the box -- Extensible architecture for custom telemetry backends -- OpenTelemetry integration following industry standards -- Performance monitoring capabilities built into the platform - -We will include the following two plugins in the initial release to demonstrate the new telemetry service: - -- Replace existing catalog implementation with `TelemetryService` -- Replace existing scaffolder implementation with `TelemetryService` - -### Non-Goals - -- Adding telemetry instrumentation to existing plugins that don't currently have it -- Only the catalog and scaffolder implementations will be refactored to use the new telemetry service. -- We will not start by providing a full Backstage SDK (aka recreate the OTEL Node SDK). -- We will not refactor the existing LoggerService. Future work to unify telemetry related concerns would be ideal, but not a goal. - -## Proposal - -Following similar patterns to other core services, we will create a new `RootTelemetryService` responsible for initializing telemetry and app-wide concerns, and the creation of plugin-specific telemetry services. - -A core concern is not initializing the telemetry service early enough in the backend lifecycle. Services that initialize before the telemetry service risk receiving no-op meters. To mitigate this, the root service will include a bootstrapping functionality that will be called as early as possible in the backend lifecycle. - -## Configuration - -All OTEL configuration will be supported via the app's `app-config.yaml` file. The config schema should be based on existing resources such as the [OTEL collector configuration](https://opentelemetry.io/docs/collector/configuration/). - -```yaml -# this is not a complete example, but it shows the general idea -telemetry: - enabled: true - - resource: - serviceName: backstage-telemetry - serviceVersion: 0.0.1 - - metrics: - enabled: true - collection: - exportIntervalMillis: 15000 - - exporters: - - type: prometheus - enabled: true - config: - port: 9464 - - tracing: - enabled: false - - instrumentations: - http: false - knex: false - express: true - # ... -``` - -## Design Details - -We will provide a thin wrapper around the OpenTelemetry Node SDK to provide a consistent interface while re-exporting the types from the SDK. This lets us provide a consistent telemetry interface while leveraging types and concepts already familiar to the community. Additionally, we will rely on the auto instrumentation features of the SDK as much as possible. - -The decision to leverage the SDK comes from the fact that OpenTelemetry is already the fundamental abstraction for metrics and tracing. - -```ts -interface TelemetryService { - createCounter(name: string, options?: MetricOptions): Counter; - // - - // provide escape hatch for advanced use cases (???) - todo: I don't like this if we don't have a good reason for it - getMeter(): Meter; -} - -interface RootTelemetryService extends TelemetryService { - forPlugin(pluginId: string): TelemetryService; - - // provide escape hatches for advanced use cases (???) - getClient(): NodeSDK; // give access to the SDK -} -``` - -### Naming Conventions - -Visit [0013-telemetry-metrics-naming-conventions](../0013-telemetry-metrics-naming-conventions/README.md) for the naming conventions. - -### Bootstrapping - -A core concern is not initializing the telemetry service early enough in the backend lifecycle and services that initialize before the telemetry service risk receiving no-op meters. To mitigate this, the root service will include a bootstrapping mechanism that will be called as early as possible in the backend lifecycle. - -Ideally, this is called at the start of the `createBackend` function [[ref](https://github.com/backstage/backstage/blob/ab539cc72bb503618df55815f24c4ba933607460/packages/backend-defaults/src/CreateBackend.ts#L73-L75)], but we need to gain access to the config file at minimum. Ideally the logger as well, but not as crucial. - -```ts -export function createBackend(): Backend { - // ... - // call the bootstrap function - Telemetry.bootstrap(); - // ... - return createSpecializedBackend({ defaultServiceFactories }); -} -``` - -### Plugins & Modules - -WIP - - - -## Release Plan - -1. Create a new `RootTelemetryService` that initializes telemetry and app-wide concerns. -1. Create the plugin-scoped `TelemetryService` that provides a telemetry service for plugins. -1. Release the telemetry service as alpha. -1. Refactor catalog and scaffolder implementations to use the new (alpha) `TelemetryService`. -1. Replace the existing setup docs by integrating the new `RootTelemetryService` to `createBackend` -1. Release the telemetry service -1. Offer a migration path for existing adopters to migrate to the new telemetry service. -1. Update all documentation to reference the new telemetry service. -1. Create follow-up action items to integrate the new telemetry service into the core system. -1. Fully deprecate all existing telemetry implementations like the existing Prometheus one-off implementations. - -## Dependencies - -1. The root telemetry service MUST BE initialized as EARLY as possible to prevent dependents from receiving no op meters, while still allowing for app-wide configuration. -1. There are one-off implementations of telemetry in the wild that may conflict with the proposed service. However, this is unlikely to be a problem as the SDK should continue to pick things up. - -## Alternatives - -1. Plugin authors continue to implement their own telemetry as they see fit. -1. Adopters continue using the [setup-opentelemetry](https://backstage.io/docs/tutorials/setup-opentelemetry/) docs as a guide. - -## Outstanding Questions - -1. Are there enough Backstage-isms to build an additional layer of abstraction for metrics and tracing on top of the OpenTelemetry SDK? Do we re-export the types from the OpenTelemetry SDK or do we create our own? How do we handle breaking changes? -1. How do we guarantee that the telemetry service is initialized early enough in the backend lifecycle, while still allowing for app-wide configuration? -1. Should we provide an escape hatch for advanced use cases? -1. Should there be a single telemetry service or should we provide more granular services (e.g. adding separate `MetricsService` and `TracingService`)? -1. What core services could leverage the auto instrumentation features of the OpenTelemetry SDK? (e.g. `DatabaseService` could leverage the `knex` auto instrumentation, `HttpService` could leverage the `express` auto instrumentation, etc.) diff --git a/beps/0013-telemetry-metrics-naming-conventions/README.md b/beps/0013-telemetry-metrics-naming-conventions/README.md deleted file mode 100644 index c6cbc85d24..0000000000 --- a/beps/0013-telemetry-metrics-naming-conventions/README.md +++ /dev/null @@ -1,113 +0,0 @@ ---- -title: OpenTelemetry Metrics Naming Conventions for Backstage -status: implementable -authors: - - '@kurtaking' -owners: - - '@kurtaking' -project-areas: - - framework - - telemetry -creation-date: 2025-06-26 ---- - -# BEP: OpenTelemetry Metrics Naming Conventions for Backstage - -- [Summary](#summary) -- [Motivation](#motivation) - - [Goals](#goals) - - [Non-Goals](#non-goals) -- [Proposal](#proposal) -- [Design Details](#design-details) -- [Release Plan](#release-plan) -- [Dependencies](#dependencies) -- [Alternatives](#alternatives) - -## Summary - -This proposal establishes standardized naming conventions for OpenTelemetry metrics across the Backstage ecosystem. It defines a hierarchical naming pattern that ensures consistency, discoverability, and compliance with OpenTelemetry semantic conventions while providing clear guidance for plugin and module authors. - -## Motivation - -Currently, Backstage has: - -- Mixed metric naming patterns: Legacy Prometheus metrics (`catalog_entities_count`) alongside newer OpenTelemetry metrics (`catalog.processing.queue.delay`) -- No standard pattern for plugin-specific metrics -- Difficulty correlating metrics across plugins and providers -- Some existing metrics don't follow OpenTelemetry semantic conventions - -We need to decided on a pattern that we can use for the telemetry service to obfuscate the underlying implementation details. - -### Goals - -- Establish consistent naming patterns across all Backstage metrics -- Ensure OpenTelemetry compliance with semantic conventions -- Provide clear guidance for plugin and service authors - -### Non-Goals - -- Breaking existing metrics immediately (backward compatibility maintained) -- Forcing all plugins to emit the same metrics -- Defining specific metric values or thresholds - -## Proposal - -Provide guidance for naming metrics for Backstage. - -## Design Details - -### Core Naming Pattern - -All Backstage metrics MUST follow this hierarchical pattern: - -`backstage.{scope}.{category}.{metric_name}` - -**Where:** - -- `backstage` is the root namespace for all Backstage metrics -- `{scope}` is the system scope (`pluginId`, `core` for core services) -- `{category}` is the logical grouping within the `{scope}` -- `{metric_name}` is the name of the metric - -### Scope - -The `scope` represents where it belongs in the Backstage ecosystem. - -- `pluginId` - A plugin-specific metric (e.g. `backstage.plugin.catalog.entities.count`) -- `core` - A core service metric (e.g. `backstage.core.database.connections.active`) - -#### Plugin Metrics - -Pattern: `backstage.{pluginId}.{category}.{metric_name}` - -```yaml -# Examples -backstage.catalog.entities.processed.total -backstage.scaffolder.tasks.completed.total -backstage.techdocs.builds.active -backstage.auth.sessions.active.total -``` - -#### Core Metrics - -Pattern: `backstage.core.{category}.{metric_name}` - -```yaml -# Examples -backstage.core.database.connections.active -backstage.core.scheduler.tasks.queued.total -backstage.core.http.requests.total -``` - -## Release Plan - -- Provide documentation for the naming conventions -- Release as part of the larger telemetry service proposal - -## Dependencies - -- [0012-telemetry-service](../0012-telemetry-service/README.md) - -## Alternatives - -Adopters implement their own metrics and naming conventions.