paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 14 Packages Projects Releases Wiki Activity

add docs on how to add custom entity providers

Browse Source 
Signed-off-by: Fredrik Adelöw <freben@gmail.com>
This commit is contained in:
Fredrik Adelöw
2022-01-25 23:05:13 +01:00
parent 5cd8efd576
commit da9cba20f2
2 changed files with 282 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/styles/vocab.txt
+3
View File
@@ -208,6 +208,7 @@ pageview
parallelization
Patrik
Peloton
performant
plantuml
Platformize
Podman
@@ -222,6 +223,7 @@ productional
Protobuf
proxying
Proxying
pubsub
pygments
pymdownx
rankdir
@@ -314,6 +316,7 @@ unmanaged
unregister
unregistration
untracked
upsert
upvote
url
URLs
docs/features/software-catalog/external-integrations.md
+279 -21
View File
@@ -9,24 +9,282 @@ Backstage natively supports importing catalog data through the use of
[entity descriptor YAML files](descriptor-format.md). However, companies that
already have an existing system for keeping track of software and its owners can
leverage those systems by integrating them with Backstage. This article shows
the most common way of doing that integration: by adding a custom catalog
_processor_.
the two common ways of doing that integration: by adding a custom catalog
_entity provider_, or by adding a _processor_.
## Background
The catalog has a frontend plugin part, that communicates via a service API to
the backend plugin part. The backend has a processing loop that repeatedly
ingests data from the sources you specify, to store them in its database.
the backend plugin part. The backend continuously ingests data from the sources
you specify, to store them in its database. The details of how this works is
detailed in [The Life of an Entity](life-of-an-entity.md). Reading that article
first is recommended.
As a Backstage adopter, you would be able to customize or extend the catalog in
several ways - by replacing the entire backend API, or by replacing entire
implementation classes at certain points in the backend, or by leveraging the
ingestion process to fetch data from your own authoritative source. Each method
has benefits and drawbacks, but this article will focus on the last one of the
above. It is the one that is the most straight forward and future proof, and
leverages a lot of benefits that come with the builtin catalog.
There are two main options for how to ingest data into the catalog: making a
[custom entity provider](#custom-entity-providers), or making a
[custom processor](#custom-processors). They both have strengths and drawbacks,
but the former would usually be preferred. Both options are presented in a
dedicated subsection below.
## Processors and the Ingestion Loop
## Custom Entity Providers
Entity providers sit at the very edge of the catalog. They are the original
sources of entities that form roots of the processing tree. The dynamic location
store API, and the static locations you can specify in your app-config, are two
examples of builtin providers in the catalog.
Some defining traits of entity providers:
- You instantiate them individually using code in your backend, and pass them to
the catalog builder. Often there's one provider instance per remote system.
- You may be responsible for actively running them. For example, some providers
need to be triggered periodically by a method call to know when they are meant
to do their job; in that case you'll have to make that happen.
- The timing of their work is entirely detached from the processing loops. One
provider may run every 30 seconds, another one on every incoming webhook call
of a certain type, etc.
- They can perform detailed updates on the set of entities that they are
responsible for. They can make full updates of the entire set, or issue
individual additions and removals.
- Their output is a set of unprocessed entities. Those are then subject to the
processing loops before becoming final, stitched entities.
- When they remove an entity, the entire subtree of processor-generated entities
under that root is eagerly removed as well.
### Creating an Entity Provider
The recommended way of instantiating the catalog backend classes is to use the
`CatalogBuilder`, as illustrated in the
[example backend here](https://github.com/backstage/backstage/blob/master/packages/backend/src/plugins/catalog.ts).
We will create a new
[`EntityProvider`](https://github.com/backstage/backstage/blob/master/plugins/catalog-backend/src/providers/types.ts)
subclass that can be added to this catalog builder.
Let's make a simple provider that can refresh a set of entities based on a
remote store. The provider part of the interface is actually tiny - you only
have to supply a (unique) name, and accept a connection from the environment
through which you can issue writes. The rest is up to the individual provider
implementation.
It is up to you where you put the code for this new processor class. For quick
experimentation you could place it in your backend package, but we recommend
putting all extensions like this in a backend plugin package of their own in the
`plugins` folder of your Backstage repo.
The class will have this basic structure:
```ts
import { UrlReader } from '@backstage/backend-common';
import { Entity } from '@backstage/catalog-model';
import {
EntityProvider,
EntityProviderConnection,
} from '@backstage/plugin-catalog-backend';
/**
* Provides entities from fictional frobs service.
*/
export class FrobsProvider implements EntityProvider {
private readonly env: string;
private readonly reader: UrlReader;
private connection?: EntityProviderConnection;
/** [1] **/
constructor(env: string, reader: UrlReader) {
this.env = env;
this.reader = reader;
}
/** [2] **/
getProviderName(): string {
return `frobs-${this.env}`;
}
/** [3] **/
async connect(connection: EntityProviderConnection): Promise<void> {
this.connection = connection;
}
/** [4] **/
async run(): Promise<void> {
if (!this.connection) {
throw new Error('Not initialized');
}
const raw = await this.reader.read(
`https://frobs-${this.env}.example.com/data`,
);
const data = JSON.parse(raw.toString());
/** [5] **/
const entities: Entity[] = frobsToEntities(data);
/** [6] **/
await this.connection.applyMutation({
type: 'full',
entities: entities.map(entity => ({
entity,
locationKey: `frobs-provider:${this.env}`,
})),
});
}
}
```
This class demonstrates several important concepts, some of which are optional.
Check out the numbered markings - let's go through them one by one.
1. The class takes an `env` parameter. This is only illustrative for the sake of
the example. We'll use this field to exhibit the type of provider where end
users may want or need to make multiple instances of the same provider, and
what the implications would be in that case.
2. The catalog requires that all registered providers return a name that is
_unique_ among those providers, and which is _stable_ over time. The reason
for these requirements is, the emitted entities for each provider instance
all hang around in a closed bucket of their own. This bucket needs to be tied
to their provider over time, and across backend restarts. We'll see below how
the processor emits some entities and what that means for its own bucket.
3. Once the catalog engine starts up, it immediately issues the `connect` call
to all known providers. This forms the bond between the code and the
database. This is also an opportunity for the provider to do one-time updates
on the connection at startup if it wants to.
4. At this point the provider contract is already complete. But the class needs
to do some actual work too! In this particular example, we chose to make a
`run` method that has to be called each time that you want to issue a sync
with the `frobs` service. Let's repeat that - this is only an example
implementation; some providers may be written in entirely different ways,
such as for example subscribing to pubsub events and only reacting to those,
or any number of other solutions. The only point is - external stimuli happen
somehow, which somehow get translated to calls on the `connection` to persist
the outcome of that. This example issues a `fetch` to the right service and
issues a full refresh of its entity bucket based on that.
5. The method translates the foreign data model to the native `Entity` form, as
expected by the catalog.
6. Finally, we issue a "mutation" to the catalog. This persists the entities in
our own bucket, along with an optional `locationKey` that's used for conflict
checks. But this is a bigger topic - mutations warrant their own explanatory
section below.
### Provider Mutations
Let's circle back to the bucket analogy.
Each provider _instance_ - not each class but each instance registered with the
catalog - has access to its own bucket of entities, and the bucket is identified
by the stable name of the provider instance. Every time the provider issues
"mutations", it changes the contents of that bucket. Nothing else outside of the
bucket is accessible.
There are two different types of mutation.
The first is `'full'`, which means to figuratively throw away the contents of
the bucket and replacing it with all of the new contents specified. Under the
hood, this is actually implemented through a highly efficient delta mechanism
for performance reasons, since it is common that the difference from one run to
the other is actually very small. This strategy is convenient for providers that
have easy access to batch-fetches of the entire subject material from a remote
source, and doesn't have access to, or does not want to compute, deltas.
The other mutation type is `'delta'`, which lets the provider explicitly upsert
or delete entities in its bucket. This mutation is convenient e.g. for event
based providers, and can also be more performant since no deltas need to be
computed, and previous bucket contents outside of the targeted set do not have
to be taken into account.
In all cases, the mutation entities are treated as _unprocessed_ entities. When
they land in the database, the registered catalog processors go to work on them
to transform them into final, processed and stitched, entities ready for
consumption.
Every entity emitted by a processor can have a `locationKey`, as shown above.
This is a critical conflict resolution key, in the form of an opaque string that
should be unique for each location that an entity could be located at, and
undefined if the entity does not have a fixed location.
In practice it should be set to the serialized location reference if the entity
is stored in Git, for example
`https://github.com/backstage/backstage/blob/master/catalog-info.yaml`, or a
similar string that distinctly pins down its origins. In our example we set it
to a string that was distinct for the provider class, plus its instance
identifying properties which in this case was the `env`.
A conflict between two entity definitions happen when they have the same entity
reference, i.e. kind, namespace, and name. In the event of a conflict, such as
if two "competing" providers try to emit entities that have the same reference
triplet, the location key will be used according to the following rules to
resolve the conflict:
- If the entity is already present in the database but does not have a location
key set, the new entity wins and will override the existing one.
- If the entity is already present in the database the new entity will only win
if the location keys of the existing and new entity are the same.
- If the entity is not already present, insert the entity into the database
along with the provided location key.
This may seem complex, but is a vital mechanism for ensuring that users aren't
permitted to do "rogue" takeovers of already registered entities that belong to
others.
## Installing the Provider
You should now be able to add this class to your backend in
`packages/backend/src/plugins/catalog.ts`:
```diff
+import { Duration } from 'luxon';
+import { FrobsProvider } from '../path/to/class';
export default async function createPlugin(
env: PluginEnvironment,
): Promise<Router> {
const builder = CatalogBuilder.create(env);
+ const frobs = new FrobsProvider('production', env.reader);
+ builder.addEntityProvider(frobs);
+ await env.scheduler.scheduleTask({
+ id: 'run_frobs_refresh',
+ fn: async () => { await frobs.run(); },
+ initialDelay: Duration.fromObject({ seconds: 10 }),
+ frequency: Duration.fromObject({ minutes: 30 }),
+ timeout: Duration.fromObject({ minutes: 10 }),
+ });
```
Note that we used the builtin scheduler facility to regularly call the `run`
method of the provider, in this example. It is a suitable driver for this
particular type of recurring task.
Start up the backend - it should now start reading from the previously
registered location and you'll see your entities start to appear in Backstage.
## Custom Processors
The other possible way of ingesting data into the catalog is through the use of
location reading catalog processors.
Processors sit in the middle of the processing loops of the catalog. They are
responsible for updating and finalizing unprocessed entities on their way to
becoming final, stitched entities. They can also, crucially, emit other entities
while doing so. Those then form branches of the entity tree.
Some defining traits of processors:
- You instantiate them using code in your backend, and pass them to the catalog
builder. There's usually only one instance of each type, which then gets
called many times over in parallel for all entities in the catalog.
- Their invocation is driven by the fixed processing loop. All processors are
unconditionally repeatedly called for all entities. You cannot control this
behavior, besides adjusting the frequency of the loop, which then applies
equally to all processors.
- They cannot control in detail the entities that they emit, the only effective
operation is upsert on their children. If they stop emitting a certain child,
that child becomes marked as an orphan; no deletions are possible.
- Their input is an unprocessed entity, and their output is modifications to
that same entity plus possibly some auxiliary data including unprocessed child
entities.
### Processors and the Ingestion Loop
The catalog holds a number of registered locations, that were added either by
site admins or by individual Backstage users. Their purpose is to reference some
@@ -54,7 +312,7 @@ added by the organization that adopts Backstage.
We will now show the process of creating a new processor and location type,
which enables the ingestion of catalog data from an existing external API.
## Deciding on the New Locations
### Deciding on the New Locations
The first step is to decide how we want to point at the system that holds our
data. Let's assume that it is internally named System-X and can be reached
@@ -87,7 +345,7 @@ If you start up the backend now, it will start to periodically say that it could
not find a processor that supports that location. So let's make a processor that
does so!
## Creating a Catalog Data Reader Processor
### Creating a Catalog Data Reader Processor
The recommended way of instantiating the catalog backend classes is to use the
`CatalogBuilder`, as illustrated in the
@@ -130,7 +388,7 @@ export class SystemXReaderProcessor implements CatalogProcessor {
try {
// Use the builtin reader facility to grab data from the
// API. If you prefer, you can just use plain fetch here
// (from the cross-fetch package), or any other method of
// (from the node-fetch package), or any other method of
// your choosing.
const data = await this.reader.read(location.target);
const json = JSON.parse(data.toString());
@@ -159,19 +417,19 @@ You should now be able to add this class to your backend in
`packages/backend/src/plugins/catalog.ts`:
```diff
+ import { SystemXReaderProcessor } from '../path/to/class';
+import { SystemXReaderProcessor } from '../path/to/class';
export default async function createPlugin(
env: PluginEnvironment,
): Promise<Router> {
const builder = new CatalogBuilder(env);
export default async function createPlugin(
env: PluginEnvironment,
): Promise<Router> {
const builder = CatalogBuilder.create(env);
+ builder.addProcessor(new SystemXReaderProcessor(env.reader));
```
Start up the backend - it should now start reading from the previously
registered location and you'll see your entities start to appear in Backstage.
## Caching processing results
### Caching processing results
The catalog periodically refreshes entities in the catalog, and in doing so it
calls out to external systems to fetch changes. This can be taxing for upstream