Merge pull request #7057 from backstage/freben/processing-doc

add the processing part of the life-of-an-entity
2021-09-04 20:58:46 +02:00
parent 7f00902d97 abe071e7f3
commit 8a076e68d6
3 changed files with 68 additions and 4 deletions
@@ -111,20 +111,78 @@ as possible.
 Each catalog deployment has a number of processors installed. They are
 responsible for receiving unprocessed entities that the catalog decided are due
 for processing, and then running that data through a number of processing
-stages. mutating the entity and emitting auxiliary data about it. When all of
+stages, mutating the entity and emitting auxiliary data about it. When all of
 that is done, the catalog takes all of that information and stores it as the
 processed entity, and errors and relations to other entities separately. Then,
 the catalog checks to see what entities are touched by that output, and triggers
 the final assembly of those (see Stitching below).

-There are several stages involved in the processing.
+![Processing overview](../../assets/features/catalog/life-of-an-entity_processing.svg)

-> TODO: More info here
+Entities are always processed one by one, but all of your catalog service hosts
+collaborate in doing so to distribute the load. Note how each processor can
+contribute to one or more of the fixed steps in the processing pipeline. First
+all of the processors' contributions to one step are run in the order that the
+processors were registered, then all of their contributions to the next step in
+the same order, and so on.
+
+Each step has the opportunity to optionally modify the entity, and to optionally
+emit other information. For example, the processor might look at information in
+the `spec` field of the entity, and emit relations that correspond to those
+declarations. If the processor emits an entity, then that entity gets stored
+verbatim with a timestamp saying that it, too, should be processed as soon as
+possible. If errors are emitted, then that signals that something is wrong with
+the entity and that it should not replace whatever previously error-free version
+we had among the final entities. If relations are emitted, then they are put in
+a dedicated relations table to be picked up by the stitching process below.
+
+> Optional low level detail note: When entities are emitted, the catalog keeps
+> track of the edges between the emitting entity and the ones emitted. This
+> happens behind the scenes, hidden from the outside, and is used to form a
+> graph. This is _not_ the same thing as relations! The purpose of these edges,
+> is to be able to detect when an entity becomes orphaned (see below), and to be
+> able to perform eager deletions throughout the graph when a root is explicitly
+> unregistered and nothing else is keeping lower nodes alive. We will talk more
+> about orphaning and deletions later on in this article.
+
+When the final step has completed, and no errors were encountered, the processed
+entity and all of the relations are finally persisted in the database. Then the
+catalog considers this entity, and all of the entities it had relations to,
+subject for stitching.
+
+It is worth noting here that the processing does not lead to deletion or
+unregistration of entities; it can only call new entities into existence or
+update entities that it has previously called into existence. More about that
+later.

 ## Stitching

+Stitching finalizes the entity, by gathering all of the output from the previous
+steps and merging them into the final object which is what is visible from the
+catalog API. As the final entity itself gets updated, the stitcher makes sure
+that the search table gets refreshed accordingly as well.
+
+> Note: The search table mentioned here is not related to the core Search
+> feature of Backstage. It's rather the table that backs the ability to filter
+> catalog API query results.
+
+![Stitching overview](../../assets/features/catalog/life-of-an-entity_stitching.svg)
+
+The diagram shows how the stitcher reads from several sources:
+
+- The processed entity, as returned from the processing step
+- The errors, if any, that were emitted by the processing step
+- All relations that were emitted by the processing step, as well as any
+  relations emitted by _other_ entity processing steps that happen to point at
+  the current entity
+
+The last part is noteworthy: This is how the stitcher is able to collect all of
+the relation edges, both incoming and outgoing, no matter who produced them.
+
 The stitching is currently a fixed process, that cannot be modified or extended.
 This means that any modifications you want to make on the final result, has to
 happen during ingestion or processing.

-> TODO: More info here
+## Deletion
+
+> TODO