paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 32 Packages Projects Releases Wiki Activity

add the processing part of the life-of-an-entity

Browse Source 
Signed-off-by: Fredrik Adelöw <freben@gmail.com>
This commit is contained in:
Fredrik Adelöw
2021-09-03 11:42:25 +02:00
parent bc8520f8cb
commit aeb7474e20
2 changed files with 43 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/features/software-catalog/life-of-an-entity.md
+40 -3
View File
@@ -111,18 +111,55 @@ as possible.
Each catalog deployment has a number of processors installed. They are
responsible for receiving unprocessed entities that the catalog decided are due
for processing, and then running that data through a number of processing
stages. mutating the entity and emitting auxiliary data about it. When all of
stages, mutating the entity and emitting auxiliary data about it. When all of
that is done, the catalog takes all of that information and stores it as the
processed entity, and errors and relations to other entities separately. Then,
the catalog checks to see what entities are touched by that output, and triggers
the final assembly of those (see Stitching below).
There are several stages involved in the processing.
![Processing overview](../../assets/features/catalog/life-of-an-entity_processing.svg)
> TODO: More info here
Entities are always processed one by one. Note how each processor can contribute
to one or more of the fixed steps in the processing pipeline. First all of the
processors' contributions to one step are run in the order that the processors
were registered, then all of their contributions to the next step in the same
order, and so on.
Each step has the opportunity to optionally modify the entity, and to optionally
emit other information. For example, the processor might look at information in
the `spec` field of the entity, and emit relations that correspond to those
declarations. If the processor emits an entity, then that entity gets stored
verbatim with a timestamp saying that it, too, should be processed as soon as
possible. If errors are emitted, then that signals that something is wrong with
the entity and that it should not replace whatever previously error-free version
we had among the final entities. If relations are emitted, then they are put in
a dedicated relations table to be picked up by the stitching process below.
> Optional low level detail note: When entities are emitted, the catalog keeps
> track of the edges between the emitting entity and the ones emitted. This
> happens behind the scenes, hidden from the outside, and is used to form a
> graph. This is _not_ the same thing as relations! The purpose of these edges,
> is to be able to detect when an entity becomes orphaned (see below), and to be
> able to perform eager deletions throughout the graph when a root is explicitly
> unregistered and nothing else is keeping lower nodes alive. We will talk more
> about orphaning and deletions later on in this article.
When the final step has completed, and no errors were encountered, the processed
entity and all of the relations are finally persisted in the database. Then the
catalog considers this entity, and all of the entities it had relations to,
subject for stitching.
It is worth noting here that the processing does not lead to deletion or
unregistration of entities; it can only call new entities into existence or
update entities that it has previously called into existence. More about that
later.
## Stitching
Stitching finalizes the entity by combining all data gathered such as relations,
errors and the processed entity itself into a final object which is essentially
the result that will be returned from the catalog API.
The stitching is currently a fixed process, that cannot be modified or extended.
This means that any modifications you want to make on the final result, has to
happen during ingestion or processing.