From aeb7474e20e513bab3d1fe7807e8d41075912c7f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fredrik=20Adel=C3=B6w?= Date: Fri, 3 Sep 2021 11:42:25 +0200 Subject: [PATCH] add the processing part of the life-of-an-entity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Fredrik Adelöw --- .../catalog/life-of-an-entity_processing.svg | 3 ++ .../software-catalog/life-of-an-entity.md | 43 +++++++++++++++++-- 2 files changed, 43 insertions(+), 3 deletions(-) create mode 100644 docs/assets/features/catalog/life-of-an-entity_processing.svg diff --git a/docs/assets/features/catalog/life-of-an-entity_processing.svg b/docs/assets/features/catalog/life-of-an-entity_processing.svg new file mode 100644 index 0000000000..da3c4c7dba --- /dev/null +++ b/docs/assets/features/catalog/life-of-an-entity_processing.svg @@ -0,0 +1,3 @@ + + +
Processor One
Processor One
Processor Two
Processor Two
Unprocessed Entity
Unprocessed Enti...
emit
e...
preprocess
preprocess

preprocess

preprocess
(intermediate)
(intermediate)
validate
validate
(intermediate)
(intermediate)

postprocess

postprocess
(intermediate)
(intermediate)
Processed
Entity
Processed...


relations,
errors,
other entities
relations,...
emit
e...


relations,
errors,
other entities
relations,...
emit
e...


relations,
errors,
other entities
relations,...
emit
e...


relations,
errors,
other entities
relations,...Viewer does not support full SVG 1.1 \ No newline at end of file diff --git a/docs/features/software-catalog/life-of-an-entity.md b/docs/features/software-catalog/life-of-an-entity.md index 817529c0f2..1d8409f9d6 100644 --- a/docs/features/software-catalog/life-of-an-entity.md +++ b/docs/features/software-catalog/life-of-an-entity.md @@ -111,18 +111,55 @@ as possible. Each catalog deployment has a number of processors installed. They are responsible for receiving unprocessed entities that the catalog decided are due for processing, and then running that data through a number of processing -stages. mutating the entity and emitting auxiliary data about it. When all of +stages, mutating the entity and emitting auxiliary data about it. When all of that is done, the catalog takes all of that information and stores it as the processed entity, and errors and relations to other entities separately. Then, the catalog checks to see what entities are touched by that output, and triggers the final assembly of those (see Stitching below). -There are several stages involved in the processing. +![Processing overview](../../assets/features/catalog/life-of-an-entity_processing.svg) -> TODO: More info here +Entities are always processed one by one. Note how each processor can contribute +to one or more of the fixed steps in the processing pipeline. First all of the +processors' contributions to one step are run in the order that the processors +were registered, then all of their contributions to the next step in the same +order, and so on. + +Each step has the opportunity to optionally modify the entity, and to optionally +emit other information. For example, the processor might look at information in +the `spec` field of the entity, and emit relations that correspond to those +declarations. If the processor emits an entity, then that entity gets stored +verbatim with a timestamp saying that it, too, should be processed as soon as +possible. If errors are emitted, then that signals that something is wrong with +the entity and that it should not replace whatever previously error-free version +we had among the final entities. If relations are emitted, then they are put in +a dedicated relations table to be picked up by the stitching process below. + +> Optional low level detail note: When entities are emitted, the catalog keeps +> track of the edges between the emitting entity and the ones emitted. This +> happens behind the scenes, hidden from the outside, and is used to form a +> graph. This is _not_ the same thing as relations! The purpose of these edges, +> is to be able to detect when an entity becomes orphaned (see below), and to be +> able to perform eager deletions throughout the graph when a root is explicitly +> unregistered and nothing else is keeping lower nodes alive. We will talk more +> about orphaning and deletions later on in this article. + +When the final step has completed, and no errors were encountered, the processed +entity and all of the relations are finally persisted in the database. Then the +catalog considers this entity, and all of the entities it had relations to, +subject for stitching. + +It is worth noting here that the processing does not lead to deletion or +unregistration of entities; it can only call new entities into existence or +update entities that it has previously called into existence. More about that +later. ## Stitching +Stitching finalizes the entity by combining all data gathered such as relations, +errors and the processed entity itself into a final object which is essentially +the result that will be returned from the catalog API. + The stitching is currently a fixed process, that cannot be modified or extended. This means that any modifications you want to make on the final result, has to happen during ingestion or processing.