more entity docs

Signed-off-by: Fredrik Adelöw <freben@gmail.com>

docs/features/software-catalog/api.md

@@ -21,7 +21,7 @@ section below is `/entities`, and the catalog is located at
 `http://localhost:7007/api/catalog` during local development, the full URL would
 be `http://localhost:7007/api/catalog/entities`. The actual URL may vary from
 one organization to the other, especially in production, but is commonly your
-`backend.baseUrl` in your app config, plus `/catalog` at the end.
+`backend.baseUrl` in your app config, plus `/api/catalog` at the end.
 
 Some or all of the endpoints may accept or require an `Authorization` header
 with a `Bearer` token, which should then be the Backstage token returned by the

docs/features/software-catalog/life-of-an-entity.md

@@ -48,9 +48,9 @@ The details of these processes are described below.
 
 ## Ingestion
 
-Each catalog deployment has a number of entity providers installed. They are
+Each catalog deployment has a number of _entity providers_ installed. They are
 responsible for fetching data from external authoritative sources in any way
-that they see fit, to translate those into entity objects, and to notify the
+that they see fit, translating those into entity objects, and notifying the
 database when those entities are added or removed. These are the _unprocessed
 entities_ that will be subject to later processing (see below), and they form
 the very basis of existence for entities. If there were no entity providers, no
@@ -108,7 +108,7 @@ the processing loop should next try to process it. When the entity first
 appears, this timestamp is set to "now" - asking for it to be picked up as soon
 as possible.
 
-Each catalog deployment has a number of processors installed. They are
+Each catalog deployment has a number of _processors_ installed. They are
 responsible for receiving unprocessed entities that the catalog decided are due
 for processing, and then running that data through a number of processing
 stages, mutating the entity and emitting auxiliary data about it. When all of
@@ -185,7 +185,28 @@ happen during ingestion or processing.
 
 ## Errors
 
-> TODO: Describe how errors are exposed through entities
+Errors during the ingestion and processing of entities can happen in a variety
+of ways, and they may happen at a far later point in time than when they were
+registered. For example, a registered file may get deleted in the remote system,
+or the user may accidentally change the file contents in such a way that they
+cannot be parsed successfully, etc.
+
+There are two main ways that these errors are surfaced.
+
+First, the catalog backend will produce detailed logs that should contain
+sufficient information for a reader to find the causes for errors. Since these
+logs are typically not easily found by end users, this can mainly be a useful
+tool for Backstage operators who want to debug problems either with statically
+registered entities that are under their control, or to help end users find
+problems.
+
+Second, for most classes of errors, the entity itself will contain a status
+field that describes the problem. The contents of this field is shown at the top
+of your entity page in Backstage, if you have placed the corresponding error
+callout component (`EntityProcessingErrorsPanel`) there.
+
+We are still working to improve the surfacing and observability around
+processing loop errors.
 
 ## Orphaning
 
@@ -206,17 +227,22 @@ either, it becomes _orphaned_. The end result is as follows:
 - The catalog page in Backstage for the child entity detects the new annotation
   and informs users about the orphan status.
 
-Orphaning can occur in several different scenarios. One common cause is that the
-end user edited a corresponding catalog catalog-info YAML file removing the
-entity's entry. In the case of a `Location` parent entity, orphaning can happen
-if removing the target line pointing to the file containing the child entity.
-Another common cause is large batch processors such as the ones that crawl
-through remote systems looking for entities, no longer finding something that it
-used to find before. Maybe the data was moved, or deleted, in the remote system.
-So for example when a person leaves the company an LDAP org discovery processor
-might leave an orphaned `User` entity behind. Note that this only applies to
-processors - ingestion that happens using entity providers work differently,
-described below.
+Orphaning can occur in several different scenarios.
+
+- If a catalog-info YAML file is moved from one place to another in the version
+  control system without updating the registration in the catalog, it will
+  effectively become orphaned "by" that registered location
+- If the user edits a corresponding parent catalog-info YAML file removing the
+  entity's entry - for example in the case of a `Location` parent entity,
+  orphaning can happen if editing or removing the `target`/`targets` lines
+  pointing to the file containing the child entity.
+- Another common cause is large batch processors such as the ones that crawl
+  through remote systems looking for entities, no longer finding something that
+  it used to find before. Maybe the data was moved, or deleted, in the remote
+  system. So for example when a person leaves the company an LDAP org discovery
+  processor might leave an orphaned `User` entity behind. Note that this only
+  applies to processors - ingestion that happens using entity providers work
+  differently, described below.
 
 > Note that removing a file, or accidentally corrupting a file so that it cannot
 > be read successfully, does _not_ lead to orphaning. Hard errors, including the
@@ -239,8 +265,78 @@ provided out of the box.
 
 ## Implicit Deletion
 
-> TODO: Describe the process of entity providers eagerly deleting entities
+Entity providers - not processors - are subject to _eager_ deletion of entities,
+which may trigger the implicit deletion of more than just the entity you thought
+you were deleting. This concept is explained here.
+
+Recall that all entity providers manage a private "bucket" of entities, as
+described in the [External integrations](external-integrations.md) article. They
+can perform some operations on those entities, including additions, updates, and
+deletions. Entity additions/updates are subject to the regular processing loops,
+which means that bucket entities may end up forming roots of an entire graph of
+entities that are emitted by those processors as they recursively work they way
+through the bucket contents and its descendants.
+
+When a provider issues a deletion of an entity in its bucket, that entity as
+well as _the entire tree of entities processed out of it_, if any, are
+considered for immediate deletion. Note "considered" - they are deleted if and
+only if they would otherwise have become orphaned (no other parent entities
+emitting them). Since the graph of entities is not strictly a tree, multiple
+roots may actually end up indirectly referencing a node farther down in the
+graph. If that's the case, that node won't go away until all such roots go away.
+
+URLs to yaml files that you register using either the Create button or add to
+your app-config, are both handled by entity providers. That means that this
+implicit deletion mechanism comes into play in some everyday circumstances.
+Let's illustrate.
+
+Imagine that you have a monorepo, with a single `Location` entity in a
+catalog-info file at the root, and that entity points to three other
+catalog-info files in the repo with a `Component` entity in each one.
+
+```text
+/
+  feature_one/
+    catalog-info.yaml     <- kind: Component
+  feature_two/
+    catalog-info.yaml     <- kind: Component
+  feature_three/
+    catalog-info.yaml     <- kind: Component
+  catalog-info.yaml       <- kind: Location
+```
+
+If you register the root `Location` entity, the actual effect is that _five_
+entities appear in the catalog. First, one that is named `generated-`-something,
+which corresponds to the registered URL itself. That's the one that the provider
+has put in its "bucket". Then, as processing loops chug along, the `Location`
+entity you pointed to appears as a child of that, and then the three `Component`
+entities appear in turn as children of the `Location`.
+
+As an end user of the Backstage interface, you may now want to delete one of the
+three `Component` entities. You do that by visiting the three-dots menu in the
+top right of an entity view. The popup dialog that appears will inform you that
+actually this entity belongs to a certain root, and that you may want to remove
+that root instead (which corresponds to unregistering the originally registered
+URL). If you choose to do so, _all_ of the aforementioned five entities will
+actually be deleted in the same operation.
+
+If you did not want to perform this aggressive pruning, you might have instead
+chosen to remove one of the `target` rows of your `Location` catalog-info file,
+and then deleted the catalog-info file that contained the `Component` you wanted
+to get rid of. Now the catalog would be left with an orphaned component, and you
+would instead be able to use the explicit deletion (see below) to delete that
+single component.
 
 ## Explicit Deletion
 
-> TODO: Describe direct deletion via the catalog API
+The catalog and its REST API also permits direct deletion of individual
+entities. This makes sense to do on orphaned entities; entities that aren't
+being actively kept up to date by any parent entities. The popup interface under
+the three-dots menu option of entity views does offer this option, and the
+orphaned status can be seen in an info box at the top of the entity's overview
+page.
+
+However, if you were to try to do an explicit depletion on an entity that's
+being kept actively updated by a parent entity, it would just reappear again
+shortly thereafter when the processing loops reconsider the parent entity that's
+still in there.