From a004b8da1a9ddf502e3eba58b666131211abbb11 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fredrik=20Adel=C3=B6w?= Date: Mon, 30 Aug 2021 07:32:58 +0200 Subject: [PATCH] add a 'life of an entity' catalog article MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Fredrik Adelöw --- .../catalog/life-of-an-entity_ingestion.svg | 3 + .../catalog/life-of-an-entity_overview.svg | 3 + .../software-catalog/life-of-an-entity.md | 130 ++++++++++++++++++ microsite/sidebars.json | 1 + mkdocs.yml | 1 + 5 files changed, 138 insertions(+) create mode 100644 docs/assets/features/catalog/life-of-an-entity_ingestion.svg create mode 100644 docs/assets/features/catalog/life-of-an-entity_overview.svg create mode 100644 docs/features/software-catalog/life-of-an-entity.md diff --git a/docs/assets/features/catalog/life-of-an-entity_ingestion.svg b/docs/assets/features/catalog/life-of-an-entity_ingestion.svg new file mode 100644 index 0000000000..b250744fcd --- /dev/null +++ b/docs/assets/features/catalog/life-of-an-entity_ingestion.svg @@ -0,0 +1,3 @@ + + +
Unprocessed Entities
Unprocessed Entities
change events
change events
Entity Provider
Entity Provider
change events
change events
Entity Provider
Entity Provider
External
Source
External...
External
Source
External...
Entity
Entity
Entity
Entity
Entity
Entity
Entity
Entity
Entity
Entity
Entity
Entity
Entity
Entity
Entity
EntityViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/docs/assets/features/catalog/life-of-an-entity_overview.svg b/docs/assets/features/catalog/life-of-an-entity_overview.svg new file mode 100644 index 0000000000..3fc35b4775 --- /dev/null +++ b/docs/assets/features/catalog/life-of-an-entity_overview.svg @@ -0,0 +1,3 @@ + + +
Unprocessed Entities
Unprocessed Entities
Processed Entities
Processed Entities
Relations
Relations
Edges
Edges
Final Entities
Final Entities
Search
Search
Errors
Errors
Processors
Processors
Stitcher
Stitcher
Entity Providers
Entity Providers
External
Sources
External...
Catalog
API
Catalog...
Processing
Processing
Stitching
Stitching
Ingestion
IngestionViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/docs/features/software-catalog/life-of-an-entity.md b/docs/features/software-catalog/life-of-an-entity.md new file mode 100644 index 0000000000..817529c0f2 --- /dev/null +++ b/docs/features/software-catalog/life-of-an-entity.md @@ -0,0 +1,130 @@ +--- +id: life-of-an-entity +title: The Life of an Entity +sidebar_label: The Life of an Entity +# prettier-ignore +description: The life cycle of entities, from being introduced into the catalog, through processing, to being removed again +--- + +This document gives a high level overview of the catalog backend, and the +technical processes involved in making entities flow through it. It is mainly +aimed at developers who want to understand the internals while installing or +extending the catalog. However, it can be informative for other personas too. + +## Key Concepts + +The catalog forms a hub of sorts, where entities are ingested from various +authoritative sources and held in a database, subject to automated processing, +and then presented through an API for quick and easy access by Backstage and +others. The most common source is [YAML files](descriptor-format.md) on a +standard format, living in version control systems near the source code of +systems that they describe. Those files are registered with the catalog and +maintained by the respective owners. The catalog makes sure to keep itself up to +date with changes to those files. + +The main extension points where developers can customize the catalog are: + +- _Entity providers_, that feed initial raw entity data into the catalog, +- _Policies_, that establish baseline rules about the shape of entities, +- _Processors_, that validate, analyze, and mutate the raw entity data into its + final form. + +The high level processes involved are: + +- _Ingestion_, where entity providers fetch raw entity data from external + sources and seed it into the database, +- _Processing_, where the policies and processors continually treat the ingested + data and may emit both other raw entities (that are also subject to + processing), errors, relations to other entities, etc., +- _Stitching_, where all of the data emitted by various processors are assembled + together into the final output entity. + +An entity is not visible to the outside world (through the catalog API), until +it has passed through the last process and landed among the final entities. + +![General overview](../../assets/features/catalog/life-of-an-entity_overview.svg) + +The details of these processes are described below. + +## Ingestion + +Each catalog deployment has a number of entity providers installed. They are +responsible for fetching data from external authoritative sources in any way +that they see fit, to translate those into entity objects, and to notify the +database when those entities are added or removed. These are the _unprocessed +entities_ that will be subject to later processing (see below), and they form +the very basis of existence for entities. If there were no entity providers, no +entities would ever enter the system. + +The database always keeps track of the set of entities that belong to each +provider; no two providers can try to output the same entity. And when a +provider signals the removal of an entity, then that leads to an _eager +deletion_: the entity and all auxiliary data that it has led to in the database +is immediately purged. + +![Ingestion overview](../../assets/features/catalog/life-of-an-entity_ingestion.svg) + +There are two providers installed by default: the one that deals with user +registered locations (e.g. URLs to YAML files), and the one that deals with +static locations in the app-config. You can add more third party providers by +passing them to the catalog builder in your backend initialization code, and you +can easily write your own. + +An entity provider is a class that implements the `EntityProvider` interface. It +has three main parts: + +- The identity: Each provider instance has a unique, stable identifier that the + database can use to keep track of the originator of each unprocessed entity. +- The connection: During backend startup, each provider is attached to the + catalog runtime. +- The stream of events: During its lifetime, the provider can issue change + events to the runtime at any point in time, to modify its set of unprocessed + entities. + +It is entirely up to the provider to choose how and when it produces these +change events. For example, the app-config provider only fires off an update at +startup and then lies dormant. The location database provider does an initial +update at startup, and then small delta updates every time a location database +change is detected. The LDAP provider is driven externally by a timer loop that +occasionally triggers a full update. Some future provider may be entirely event +driven, feeding off an event bus or web hook. There is no magic coordination +among providers; if they need to arrange synchronization or locking among +themselves for example to avoid duplicate work across multiple catalog service +machines, they need to handle that out-of-band. + +The entities that are emitted get some coarse validation applied to them, to +ensure that they at least adhere to the most basic schema rules about how an +entity should be shaped. For example, they need to have a `kind`, a +`metadata.name`, and optionally a `metadata.namespace`, among others. Apart from +that, the ingestion stage considers its work done, and stores the unprocessed +entities to be picked up at a later time by the processing system. This means +that the more precise validation rules that you put in place on entities are +_not_ yet applied at this stage. + +## Processing + +Every unprocessed entity comes with a timestamp, which tells at what time that +the processing loop should next try to process it. When the entity first +appears, this timestamp is set to "now" - asking for it to be picked up as soon +as possible. + +Each catalog deployment has a number of processors installed. They are +responsible for receiving unprocessed entities that the catalog decided are due +for processing, and then running that data through a number of processing +stages. mutating the entity and emitting auxiliary data about it. When all of +that is done, the catalog takes all of that information and stores it as the +processed entity, and errors and relations to other entities separately. Then, +the catalog checks to see what entities are touched by that output, and triggers +the final assembly of those (see Stitching below). + +There are several stages involved in the processing. + +> TODO: More info here + +## Stitching + +The stitching is currently a fixed process, that cannot be modified or extended. +This means that any modifications you want to make on the final result, has to +happen during ingestion or processing. + +> TODO: More info here diff --git a/microsite/sidebars.json b/microsite/sidebars.json index d787fb28aa..d200b615f1 100644 --- a/microsite/sidebars.json +++ b/microsite/sidebars.json @@ -36,6 +36,7 @@ "label": "Software Catalog", "ids": [ "features/software-catalog/software-catalog-overview", + "features/software-catalog/life-of-an-entity", "features/software-catalog/configuration", "features/software-catalog/system-model", "features/software-catalog/descriptor-format", diff --git a/mkdocs.yml b/mkdocs.yml index 73ad52341e..bddce144d3 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -33,6 +33,7 @@ nav: - Core Features: - Software Catalog: - Overview: 'features/software-catalog/index.md' + - The Life of an Entity: 'features/software-catalog/life-of-an-entity.md' - Catalog Configuration: 'features/software-catalog/configuration.md' - System Model: 'features/software-catalog/system-model.md' - YAML File Format: 'features/software-catalog/descriptor-format.md'