From aeb7474e20e513bab3d1fe7807e8d41075912c7f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fredrik=20Adel=C3=B6w?= Date: Fri, 3 Sep 2021 11:42:25 +0200 Subject: [PATCH 1/3] add the processing part of the life-of-an-entity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Fredrik Adelöw --- .../catalog/life-of-an-entity_processing.svg | 3 ++ .../software-catalog/life-of-an-entity.md | 43 +++++++++++++++++-- 2 files changed, 43 insertions(+), 3 deletions(-) create mode 100644 docs/assets/features/catalog/life-of-an-entity_processing.svg diff --git a/docs/assets/features/catalog/life-of-an-entity_processing.svg b/docs/assets/features/catalog/life-of-an-entity_processing.svg new file mode 100644 index 0000000000..da3c4c7dba --- /dev/null +++ b/docs/assets/features/catalog/life-of-an-entity_processing.svg @@ -0,0 +1,3 @@ + + +
Processor One
Processor One
Processor Two
Processor Two
Unprocessed Entity
Unprocessed Enti...
emit
e...
preprocess
preprocess

preprocess

preprocess
(intermediate)
(intermediate)
validate
validate
(intermediate)
(intermediate)

postprocess

postprocess
(intermediate)
(intermediate)
Processed
Entity
Processed...


relations,
errors,
other entities
relations,...
emit
e...


relations,
errors,
other entities
relations,...
emit
e...


relations,
errors,
other entities
relations,...
emit
e...


relations,
errors,
other entities
relations,...Viewer does not support full SVG 1.1 \ No newline at end of file diff --git a/docs/features/software-catalog/life-of-an-entity.md b/docs/features/software-catalog/life-of-an-entity.md index 817529c0f2..1d8409f9d6 100644 --- a/docs/features/software-catalog/life-of-an-entity.md +++ b/docs/features/software-catalog/life-of-an-entity.md @@ -111,18 +111,55 @@ as possible. Each catalog deployment has a number of processors installed. They are responsible for receiving unprocessed entities that the catalog decided are due for processing, and then running that data through a number of processing -stages. mutating the entity and emitting auxiliary data about it. When all of +stages, mutating the entity and emitting auxiliary data about it. When all of that is done, the catalog takes all of that information and stores it as the processed entity, and errors and relations to other entities separately. Then, the catalog checks to see what entities are touched by that output, and triggers the final assembly of those (see Stitching below). -There are several stages involved in the processing. +![Processing overview](../../assets/features/catalog/life-of-an-entity_processing.svg) -> TODO: More info here +Entities are always processed one by one. Note how each processor can contribute +to one or more of the fixed steps in the processing pipeline. First all of the +processors' contributions to one step are run in the order that the processors +were registered, then all of their contributions to the next step in the same +order, and so on. + +Each step has the opportunity to optionally modify the entity, and to optionally +emit other information. For example, the processor might look at information in +the `spec` field of the entity, and emit relations that correspond to those +declarations. If the processor emits an entity, then that entity gets stored +verbatim with a timestamp saying that it, too, should be processed as soon as +possible. If errors are emitted, then that signals that something is wrong with +the entity and that it should not replace whatever previously error-free version +we had among the final entities. If relations are emitted, then they are put in +a dedicated relations table to be picked up by the stitching process below. + +> Optional low level detail note: When entities are emitted, the catalog keeps +> track of the edges between the emitting entity and the ones emitted. This +> happens behind the scenes, hidden from the outside, and is used to form a +> graph. This is _not_ the same thing as relations! The purpose of these edges, +> is to be able to detect when an entity becomes orphaned (see below), and to be +> able to perform eager deletions throughout the graph when a root is explicitly +> unregistered and nothing else is keeping lower nodes alive. We will talk more +> about orphaning and deletions later on in this article. + +When the final step has completed, and no errors were encountered, the processed +entity and all of the relations are finally persisted in the database. Then the +catalog considers this entity, and all of the entities it had relations to, +subject for stitching. + +It is worth noting here that the processing does not lead to deletion or +unregistration of entities; it can only call new entities into existence or +update entities that it has previously called into existence. More about that +later. ## Stitching +Stitching finalizes the entity by combining all data gathered such as relations, +errors and the processed entity itself into a final object which is essentially +the result that will be returned from the catalog API. + The stitching is currently a fixed process, that cannot be modified or extended. This means that any modifications you want to make on the final result, has to happen during ingestion or processing. From abecec91a6a208c21f63adcfccf0d82ef25f3bb5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fredrik=20Adel=C3=B6w?= Date: Fri, 3 Sep 2021 14:46:45 +0200 Subject: [PATCH 2/3] highlight distributed processing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Fredrik Adelöw --- docs/features/software-catalog/life-of-an-entity.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/features/software-catalog/life-of-an-entity.md b/docs/features/software-catalog/life-of-an-entity.md index 1d8409f9d6..3199ecab0a 100644 --- a/docs/features/software-catalog/life-of-an-entity.md +++ b/docs/features/software-catalog/life-of-an-entity.md @@ -119,11 +119,12 @@ the final assembly of those (see Stitching below). ![Processing overview](../../assets/features/catalog/life-of-an-entity_processing.svg) -Entities are always processed one by one. Note how each processor can contribute -to one or more of the fixed steps in the processing pipeline. First all of the -processors' contributions to one step are run in the order that the processors -were registered, then all of their contributions to the next step in the same -order, and so on. +Entities are always processed one by one, but all of your catalog service hosts +collaborate in doing so to distribute the load. Note how each processor can +contribute to one or more of the fixed steps in the processing pipeline. First +all of the processors' contributions to one step are run in the order that the +processors were registered, then all of their contributions to the next step in +the same order, and so on. Each step has the opportunity to optionally modify the entity, and to optionally emit other information. For example, the processor might look at information in From abe071e7f3aba064f40beb60b68119ca04400bbe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fredrik=20Adel=C3=B6w?= Date: Sat, 4 Sep 2021 16:30:43 +0200 Subject: [PATCH 3/3] do stitching as well MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Fredrik Adelöw --- .../catalog/life-of-an-entity_stitching.svg | 3 ++ .../software-catalog/life-of-an-entity.md | 28 ++++++++++++++++--- 2 files changed, 27 insertions(+), 4 deletions(-) create mode 100644 docs/assets/features/catalog/life-of-an-entity_stitching.svg diff --git a/docs/assets/features/catalog/life-of-an-entity_stitching.svg b/docs/assets/features/catalog/life-of-an-entity_stitching.svg new file mode 100644 index 0000000000..3b232a361c --- /dev/null +++ b/docs/assets/features/catalog/life-of-an-entity_stitching.svg @@ -0,0 +1,3 @@ + + +
Processed
Entity
Processed...
Processed
Entity
Processed...
Processed
Entity
Processed...
Final
Entity
Final...
Errors
Errors
Errors
Errors
Errors
Errors
Stitcher
Stitcher
Relation
Relation
Relation
Relation
Relation
Relation
Relation
Relation
Relation
Relation
Search
SearchViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/docs/features/software-catalog/life-of-an-entity.md b/docs/features/software-catalog/life-of-an-entity.md index 3199ecab0a..3b7b3e8117 100644 --- a/docs/features/software-catalog/life-of-an-entity.md +++ b/docs/features/software-catalog/life-of-an-entity.md @@ -157,12 +157,32 @@ later. ## Stitching -Stitching finalizes the entity by combining all data gathered such as relations, -errors and the processed entity itself into a final object which is essentially -the result that will be returned from the catalog API. +Stitching finalizes the entity, by gathering all of the output from the previous +steps and merging them into the final object which is what is visible from the +catalog API. As the final entity itself gets updated, the stitcher makes sure +that the search table gets refreshed accordingly as well. + +> Note: The search table mentioned here is not related to the core Search +> feature of Backstage. It's rather the table that backs the ability to filter +> catalog API query results. + +![Stitching overview](../../assets/features/catalog/life-of-an-entity_stitching.svg) + +The diagram shows how the stitcher reads from several sources: + +- The processed entity, as returned from the processing step +- The errors, if any, that were emitted by the processing step +- All relations that were emitted by the processing step, as well as any + relations emitted by _other_ entity processing steps that happen to point at + the current entity + +The last part is noteworthy: This is how the stitcher is able to collect all of +the relation edges, both incoming and outgoing, no matter who produced them. The stitching is currently a fixed process, that cannot be modified or extended. This means that any modifications you want to make on the final result, has to happen during ingestion or processing. -> TODO: More info here +## Deletion + +> TODO