From 85e5bdfe5109737c34b9a743eef0790198661386 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fredrik=20Adel=C3=B6w?= Date: Thu, 3 Feb 2022 13:33:54 +0100 Subject: [PATCH] more entity docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Fredrik Adelöw --- .github/styles/vocab.txt | 4 +- docs/features/software-catalog/api.md | 2 +- .../software-catalog/life-of-an-entity.md | 130 +++++++++++++++--- 3 files changed, 117 insertions(+), 19 deletions(-) diff --git a/.github/styles/vocab.txt b/.github/styles/vocab.txt index 3abe840368..8303510e77 100644 --- a/.github/styles/vocab.txt +++ b/.github/styles/vocab.txt @@ -28,6 +28,7 @@ Blackbox bool boolean builtins +callout Chai changeset changesets @@ -315,6 +316,7 @@ unbreak Unconference unmanaged unregister +unregistering unregistration untracked upsert @@ -338,4 +340,4 @@ Zalando Zhou zoomable zsh -Alef \ No newline at end of file +Alef diff --git a/docs/features/software-catalog/api.md b/docs/features/software-catalog/api.md index 71e229651a..3221b9e46a 100644 --- a/docs/features/software-catalog/api.md +++ b/docs/features/software-catalog/api.md @@ -21,7 +21,7 @@ section below is `/entities`, and the catalog is located at `http://localhost:7007/api/catalog` during local development, the full URL would be `http://localhost:7007/api/catalog/entities`. The actual URL may vary from one organization to the other, especially in production, but is commonly your -`backend.baseUrl` in your app config, plus `/catalog` at the end. +`backend.baseUrl` in your app config, plus `/api/catalog` at the end. Some or all of the endpoints may accept or require an `Authorization` header with a `Bearer` token, which should then be the Backstage token returned by the diff --git a/docs/features/software-catalog/life-of-an-entity.md b/docs/features/software-catalog/life-of-an-entity.md index 9e8a3f0563..618923289e 100644 --- a/docs/features/software-catalog/life-of-an-entity.md +++ b/docs/features/software-catalog/life-of-an-entity.md @@ -48,9 +48,9 @@ The details of these processes are described below. ## Ingestion -Each catalog deployment has a number of entity providers installed. They are +Each catalog deployment has a number of _entity providers_ installed. They are responsible for fetching data from external authoritative sources in any way -that they see fit, to translate those into entity objects, and to notify the +that they see fit, translating those into entity objects, and notifying the database when those entities are added or removed. These are the _unprocessed entities_ that will be subject to later processing (see below), and they form the very basis of existence for entities. If there were no entity providers, no @@ -108,7 +108,7 @@ the processing loop should next try to process it. When the entity first appears, this timestamp is set to "now" - asking for it to be picked up as soon as possible. -Each catalog deployment has a number of processors installed. They are +Each catalog deployment has a number of _processors_ installed. They are responsible for receiving unprocessed entities that the catalog decided are due for processing, and then running that data through a number of processing stages, mutating the entity and emitting auxiliary data about it. When all of @@ -185,7 +185,28 @@ happen during ingestion or processing. ## Errors -> TODO: Describe how errors are exposed through entities +Errors during the ingestion and processing of entities can happen in a variety +of ways, and they may happen at a far later point in time than when they were +registered. For example, a registered file may get deleted in the remote system, +or the user may accidentally change the file contents in such a way that they +cannot be parsed successfully, etc. + +There are two main ways that these errors are surfaced. + +First, the catalog backend will produce detailed logs that should contain +sufficient information for a reader to find the causes for errors. Since these +logs are typically not easily found by end users, this can mainly be a useful +tool for Backstage operators who want to debug problems either with statically +registered entities that are under their control, or to help end users find +problems. + +Second, for most classes of errors, the entity itself will contain a status +field that describes the problem. The contents of this field is shown at the top +of your entity page in Backstage, if you have placed the corresponding error +callout component (`EntityProcessingErrorsPanel`) there. + +We are still working to improve the surfacing and observability around +processing loop errors. ## Orphaning @@ -206,17 +227,22 @@ either, it becomes _orphaned_. The end result is as follows: - The catalog page in Backstage for the child entity detects the new annotation and informs users about the orphan status. -Orphaning can occur in several different scenarios. One common cause is that the -end user edited a corresponding catalog catalog-info YAML file removing the -entity's entry. In the case of a `Location` parent entity, orphaning can happen -if removing the target line pointing to the file containing the child entity. -Another common cause is large batch processors such as the ones that crawl -through remote systems looking for entities, no longer finding something that it -used to find before. Maybe the data was moved, or deleted, in the remote system. -So for example when a person leaves the company an LDAP org discovery processor -might leave an orphaned `User` entity behind. Note that this only applies to -processors - ingestion that happens using entity providers work differently, -described below. +Orphaning can occur in several different scenarios. + +- If a catalog-info YAML file is moved from one place to another in the version + control system without updating the registration in the catalog, it will + effectively become orphaned "by" that registered location +- If the user edits a corresponding parent catalog-info YAML file removing the + entity's entry - for example in the case of a `Location` parent entity, + orphaning can happen if editing or removing the `target`/`targets` lines + pointing to the file containing the child entity. +- Another common cause is large batch processors such as the ones that crawl + through remote systems looking for entities, no longer finding something that + it used to find before. Maybe the data was moved, or deleted, in the remote + system. So for example when a person leaves the company an LDAP org discovery + processor might leave an orphaned `User` entity behind. Note that this only + applies to processors - ingestion that happens using entity providers work + differently, described below. > Note that removing a file, or accidentally corrupting a file so that it cannot > be read successfully, does _not_ lead to orphaning. Hard errors, including the @@ -239,8 +265,78 @@ provided out of the box. ## Implicit Deletion -> TODO: Describe the process of entity providers eagerly deleting entities +Entity providers - not processors - are subject to _eager_ deletion of entities, +which may trigger the implicit deletion of more than just the entity you thought +you were deleting. This concept is explained here. + +Recall that all entity providers manage a private "bucket" of entities, as +described in the [External integrations](external-integrations.md) article. They +can perform some operations on those entities, including additions, updates, and +deletions. Entity additions/updates are subject to the regular processing loops, +which means that bucket entities may end up forming roots of an entire graph of +entities that are emitted by those processors as they recursively work they way +through the bucket contents and its descendants. + +When a provider issues a deletion of an entity in its bucket, that entity as +well as _the entire tree of entities processed out of it_, if any, are +considered for immediate deletion. Note "considered" - they are deleted if and +only if they would otherwise have become orphaned (no other parent entities +emitting them). Since the graph of entities is not strictly a tree, multiple +roots may actually end up indirectly referencing a node farther down in the +graph. If that's the case, that node won't go away until all such roots go away. + +URLs to yaml files that you register using either the Create button or add to +your app-config, are both handled by entity providers. That means that this +implicit deletion mechanism comes into play in some everyday circumstances. +Let's illustrate. + +Imagine that you have a monorepo, with a single `Location` entity in a +catalog-info file at the root, and that entity points to three other +catalog-info files in the repo with a `Component` entity in each one. + +```text +/ + feature_one/ + catalog-info.yaml <- kind: Component + feature_two/ + catalog-info.yaml <- kind: Component + feature_three/ + catalog-info.yaml <- kind: Component + catalog-info.yaml <- kind: Location +``` + +If you register the root `Location` entity, the actual effect is that _five_ +entities appear in the catalog. First, one that is named `generated-`-something, +which corresponds to the registered URL itself. That's the one that the provider +has put in its "bucket". Then, as processing loops chug along, the `Location` +entity you pointed to appears as a child of that, and then the three `Component` +entities appear in turn as children of the `Location`. + +As an end user of the Backstage interface, you may now want to delete one of the +three `Component` entities. You do that by visiting the three-dots menu in the +top right of an entity view. The popup dialog that appears will inform you that +actually this entity belongs to a certain root, and that you may want to remove +that root instead (which corresponds to unregistering the originally registered +URL). If you choose to do so, _all_ of the aforementioned five entities will +actually be deleted in the same operation. + +If you did not want to perform this aggressive pruning, you might have instead +chosen to remove one of the `target` rows of your `Location` catalog-info file, +and then deleted the catalog-info file that contained the `Component` you wanted +to get rid of. Now the catalog would be left with an orphaned component, and you +would instead be able to use the explicit deletion (see below) to delete that +single component. ## Explicit Deletion -> TODO: Describe direct deletion via the catalog API +The catalog and its REST API also permits direct deletion of individual +entities. This makes sense to do on orphaned entities; entities that aren't +being actively kept up to date by any parent entities. The popup interface under +the three-dots menu option of entity views does offer this option, and the +orphaned status can be seen in an info box at the top of the entity's overview +page. + +However, if you were to try to do an explicit depletion on an entity that's +being kept actively updated by a parent entity, it would just reappear again +shortly thereafter when the processing loops reconsider the parent entity that's +still in there.