diff --git a/.github/styles/vocab.txt b/.github/styles/vocab.txt index c952b6a985..dbc84f0a58 100644 --- a/.github/styles/vocab.txt +++ b/.github/styles/vocab.txt @@ -208,6 +208,7 @@ pageview parallelization Patrik Peloton +performant plantuml Platformize Podman @@ -222,6 +223,7 @@ productional Protobuf proxying Proxying +pubsub pygments pymdownx rankdir @@ -314,6 +316,7 @@ unmanaged unregister unregistration untracked +upsert upvote url URLs diff --git a/docs/features/software-catalog/external-integrations.md b/docs/features/software-catalog/external-integrations.md index 3fec28f176..6a786d84d0 100644 --- a/docs/features/software-catalog/external-integrations.md +++ b/docs/features/software-catalog/external-integrations.md @@ -9,24 +9,282 @@ Backstage natively supports importing catalog data through the use of [entity descriptor YAML files](descriptor-format.md). However, companies that already have an existing system for keeping track of software and its owners can leverage those systems by integrating them with Backstage. This article shows -the most common way of doing that integration: by adding a custom catalog -_processor_. +the two common ways of doing that integration: by adding a custom catalog +_entity provider_, or by adding a _processor_. ## Background The catalog has a frontend plugin part, that communicates via a service API to -the backend plugin part. The backend has a processing loop that repeatedly -ingests data from the sources you specify, to store them in its database. +the backend plugin part. The backend continuously ingests data from the sources +you specify, to store them in its database. The details of how this works is +detailed in [The Life of an Entity](life-of-an-entity.md). Reading that article +first is recommended. -As a Backstage adopter, you would be able to customize or extend the catalog in -several ways - by replacing the entire backend API, or by replacing entire -implementation classes at certain points in the backend, or by leveraging the -ingestion process to fetch data from your own authoritative source. Each method -has benefits and drawbacks, but this article will focus on the last one of the -above. It is the one that is the most straight forward and future proof, and -leverages a lot of benefits that come with the builtin catalog. +There are two main options for how to ingest data into the catalog: making a +[custom entity provider](#custom-entity-providers), or making a +[custom processor](#custom-processors). They both have strengths and drawbacks, +but the former would usually be preferred. Both options are presented in a +dedicated subsection below. -## Processors and the Ingestion Loop +## Custom Entity Providers + +Entity providers sit at the very edge of the catalog. They are the original +sources of entities that form roots of the processing tree. The dynamic location +store API, and the static locations you can specify in your app-config, are two +examples of builtin providers in the catalog. + +Some defining traits of entity providers: + +- You instantiate them individually using code in your backend, and pass them to + the catalog builder. Often there's one provider instance per remote system. +- You may be responsible for actively running them. For example, some providers + need to be triggered periodically by a method call to know when they are meant + to do their job; in that case you'll have to make that happen. +- The timing of their work is entirely detached from the processing loops. One + provider may run every 30 seconds, another one on every incoming webhook call + of a certain type, etc. +- They can perform detailed updates on the set of entities that they are + responsible for. They can make full updates of the entire set, or issue + individual additions and removals. +- Their output is a set of unprocessed entities. Those are then subject to the + processing loops before becoming final, stitched entities. +- When they remove an entity, the entire subtree of processor-generated entities + under that root is eagerly removed as well. + +### Creating an Entity Provider + +The recommended way of instantiating the catalog backend classes is to use the +`CatalogBuilder`, as illustrated in the +[example backend here](https://github.com/backstage/backstage/blob/master/packages/backend/src/plugins/catalog.ts). +We will create a new +[`EntityProvider`](https://github.com/backstage/backstage/blob/master/plugins/catalog-backend/src/providers/types.ts) +subclass that can be added to this catalog builder. + +Let's make a simple provider that can refresh a set of entities based on a +remote store. The provider part of the interface is actually tiny - you only +have to supply a (unique) name, and accept a connection from the environment +through which you can issue writes. The rest is up to the individual provider +implementation. + +It is up to you where you put the code for this new processor class. For quick +experimentation you could place it in your backend package, but we recommend +putting all extensions like this in a backend plugin package of their own in the +`plugins` folder of your Backstage repo. + +The class will have this basic structure: + +```ts +import { UrlReader } from '@backstage/backend-common'; +import { Entity } from '@backstage/catalog-model'; +import { + EntityProvider, + EntityProviderConnection, +} from '@backstage/plugin-catalog-backend'; + +/** + * Provides entities from fictional frobs service. + */ +export class FrobsProvider implements EntityProvider { + private readonly env: string; + private readonly reader: UrlReader; + private connection?: EntityProviderConnection; + + /** [1] **/ + constructor(env: string, reader: UrlReader) { + this.env = env; + this.reader = reader; + } + + /** [2] **/ + getProviderName(): string { + return `frobs-${this.env}`; + } + + /** [3] **/ + async connect(connection: EntityProviderConnection): Promise { + this.connection = connection; + } + + /** [4] **/ + async run(): Promise { + if (!this.connection) { + throw new Error('Not initialized'); + } + + const raw = await this.reader.read( + `https://frobs-${this.env}.example.com/data`, + ); + const data = JSON.parse(raw.toString()); + + /** [5] **/ + const entities: Entity[] = frobsToEntities(data); + + /** [6] **/ + await this.connection.applyMutation({ + type: 'full', + entities: entities.map(entity => ({ + entity, + locationKey: `frobs-provider:${this.env}`, + })), + }); + } +} +``` + +This class demonstrates several important concepts, some of which are optional. +Check out the numbered markings - let's go through them one by one. + +1. The class takes an `env` parameter. This is only illustrative for the sake of + the example. We'll use this field to exhibit the type of provider where end + users may want or need to make multiple instances of the same provider, and + what the implications would be in that case. +2. The catalog requires that all registered providers return a name that is + _unique_ among those providers, and which is _stable_ over time. The reason + for these requirements is, the emitted entities for each provider instance + all hang around in a closed bucket of their own. This bucket needs to be tied + to their provider over time, and across backend restarts. We'll see below how + the processor emits some entities and what that means for its own bucket. +3. Once the catalog engine starts up, it immediately issues the `connect` call + to all known providers. This forms the bond between the code and the + database. This is also an opportunity for the provider to do one-time updates + on the connection at startup if it wants to. +4. At this point the provider contract is already complete. But the class needs + to do some actual work too! In this particular example, we chose to make a + `run` method that has to be called each time that you want to issue a sync + with the `frobs` service. Let's repeat that - this is only an example + implementation; some providers may be written in entirely different ways, + such as for example subscribing to pubsub events and only reacting to those, + or any number of other solutions. The only point is - external stimuli happen + somehow, which somehow get translated to calls on the `connection` to persist + the outcome of that. This example issues a `fetch` to the right service and + issues a full refresh of its entity bucket based on that. +5. The method translates the foreign data model to the native `Entity` form, as + expected by the catalog. +6. Finally, we issue a "mutation" to the catalog. This persists the entities in + our own bucket, along with an optional `locationKey` that's used for conflict + checks. But this is a bigger topic - mutations warrant their own explanatory + section below. + +### Provider Mutations + +Let's circle back to the bucket analogy. + +Each provider _instance_ - not each class but each instance registered with the +catalog - has access to its own bucket of entities, and the bucket is identified +by the stable name of the provider instance. Every time the provider issues +"mutations", it changes the contents of that bucket. Nothing else outside of the +bucket is accessible. + +There are two different types of mutation. + +The first is `'full'`, which means to figuratively throw away the contents of +the bucket and replacing it with all of the new contents specified. Under the +hood, this is actually implemented through a highly efficient delta mechanism +for performance reasons, since it is common that the difference from one run to +the other is actually very small. This strategy is convenient for providers that +have easy access to batch-fetches of the entire subject material from a remote +source, and doesn't have access to, or does not want to compute, deltas. + +The other mutation type is `'delta'`, which lets the provider explicitly upsert +or delete entities in its bucket. This mutation is convenient e.g. for event +based providers, and can also be more performant since no deltas need to be +computed, and previous bucket contents outside of the targeted set do not have +to be taken into account. + +In all cases, the mutation entities are treated as _unprocessed_ entities. When +they land in the database, the registered catalog processors go to work on them +to transform them into final, processed and stitched, entities ready for +consumption. + +Every entity emitted by a processor can have a `locationKey`, as shown above. +This is a critical conflict resolution key, in the form of an opaque string that +should be unique for each location that an entity could be located at, and +undefined if the entity does not have a fixed location. + +In practice it should be set to the serialized location reference if the entity +is stored in Git, for example +`https://github.com/backstage/backstage/blob/master/catalog-info.yaml`, or a +similar string that distinctly pins down its origins. In our example we set it +to a string that was distinct for the provider class, plus its instance +identifying properties which in this case was the `env`. + +A conflict between two entity definitions happen when they have the same entity +reference, i.e. kind, namespace, and name. In the event of a conflict, such as +if two "competing" providers try to emit entities that have the same reference +triplet, the location key will be used according to the following rules to +resolve the conflict: + +- If the entity is already present in the database but does not have a location + key set, the new entity wins and will override the existing one. +- If the entity is already present in the database the new entity will only win + if the location keys of the existing and new entity are the same. +- If the entity is not already present, insert the entity into the database + along with the provided location key. + +This may seem complex, but is a vital mechanism for ensuring that users aren't +permitted to do "rogue" takeovers of already registered entities that belong to +others. + +## Installing the Provider + +You should now be able to add this class to your backend in +`packages/backend/src/plugins/catalog.ts`: + +```diff ++import { Duration } from 'luxon'; ++import { FrobsProvider } from '../path/to/class'; + + export default async function createPlugin( + env: PluginEnvironment, + ): Promise { + const builder = CatalogBuilder.create(env); + ++ const frobs = new FrobsProvider('production', env.reader); ++ builder.addEntityProvider(frobs); + ++ await env.scheduler.scheduleTask({ ++ id: 'run_frobs_refresh', ++ fn: async () => { await frobs.run(); }, ++ initialDelay: Duration.fromObject({ seconds: 10 }), ++ frequency: Duration.fromObject({ minutes: 30 }), ++ timeout: Duration.fromObject({ minutes: 10 }), ++ }); +``` + +Note that we used the builtin scheduler facility to regularly call the `run` +method of the provider, in this example. It is a suitable driver for this +particular type of recurring task. + +Start up the backend - it should now start reading from the previously +registered location and you'll see your entities start to appear in Backstage. + +## Custom Processors + +The other possible way of ingesting data into the catalog is through the use of +location reading catalog processors. + +Processors sit in the middle of the processing loops of the catalog. They are +responsible for updating and finalizing unprocessed entities on their way to +becoming final, stitched entities. They can also, crucially, emit other entities +while doing so. Those then form branches of the entity tree. + +Some defining traits of processors: + +- You instantiate them using code in your backend, and pass them to the catalog + builder. There's usually only one instance of each type, which then gets + called many times over in parallel for all entities in the catalog. +- Their invocation is driven by the fixed processing loop. All processors are + unconditionally repeatedly called for all entities. You cannot control this + behavior, besides adjusting the frequency of the loop, which then applies + equally to all processors. +- They cannot control in detail the entities that they emit, the only effective + operation is upsert on their children. If they stop emitting a certain child, + that child becomes marked as an orphan; no deletions are possible. +- Their input is an unprocessed entity, and their output is modifications to + that same entity plus possibly some auxiliary data including unprocessed child + entities. + +### Processors and the Ingestion Loop The catalog holds a number of registered locations, that were added either by site admins or by individual Backstage users. Their purpose is to reference some @@ -54,7 +312,7 @@ added by the organization that adopts Backstage. We will now show the process of creating a new processor and location type, which enables the ingestion of catalog data from an existing external API. -## Deciding on the New Locations +### Deciding on the New Locations The first step is to decide how we want to point at the system that holds our data. Let's assume that it is internally named System-X and can be reached @@ -87,7 +345,7 @@ If you start up the backend now, it will start to periodically say that it could not find a processor that supports that location. So let's make a processor that does so! -## Creating a Catalog Data Reader Processor +### Creating a Catalog Data Reader Processor The recommended way of instantiating the catalog backend classes is to use the `CatalogBuilder`, as illustrated in the @@ -130,7 +388,7 @@ export class SystemXReaderProcessor implements CatalogProcessor { try { // Use the builtin reader facility to grab data from the // API. If you prefer, you can just use plain fetch here - // (from the cross-fetch package), or any other method of + // (from the node-fetch package), or any other method of // your choosing. const data = await this.reader.read(location.target); const json = JSON.parse(data.toString()); @@ -159,19 +417,19 @@ You should now be able to add this class to your backend in `packages/backend/src/plugins/catalog.ts`: ```diff -+ import { SystemXReaderProcessor } from '../path/to/class'; ++import { SystemXReaderProcessor } from '../path/to/class'; -export default async function createPlugin( - env: PluginEnvironment, -): Promise { - const builder = new CatalogBuilder(env); + export default async function createPlugin( + env: PluginEnvironment, + ): Promise { + const builder = CatalogBuilder.create(env); + builder.addProcessor(new SystemXReaderProcessor(env.reader)); ``` Start up the backend - it should now start reading from the previously registered location and you'll see your entities start to appear in Backstage. -## Caching processing results +### Caching processing results The catalog periodically refreshes entities in the catalog, and in doing so it calls out to external systems to fetch changes. This can be taxing for upstream