|
|
|
@@ -48,9 +48,9 @@ The details of these processes are described below.
|
|
|
|
|
|
|
|
|
|
## Ingestion
|
|
|
|
|
|
|
|
|
|
Each catalog deployment has a number of entity providers installed. They are
|
|
|
|
|
Each catalog deployment has a number of _entity providers_ installed. They are
|
|
|
|
|
responsible for fetching data from external authoritative sources in any way
|
|
|
|
|
that they see fit, to translate those into entity objects, and to notify the
|
|
|
|
|
that they see fit, translating those into entity objects, and notifying the
|
|
|
|
|
database when those entities are added or removed. These are the _unprocessed
|
|
|
|
|
entities_ that will be subject to later processing (see below), and they form
|
|
|
|
|
the very basis of existence for entities. If there were no entity providers, no
|
|
|
|
@@ -108,7 +108,7 @@ the processing loop should next try to process it. When the entity first
|
|
|
|
|
appears, this timestamp is set to "now" - asking for it to be picked up as soon
|
|
|
|
|
as possible.
|
|
|
|
|
|
|
|
|
|
Each catalog deployment has a number of processors installed. They are
|
|
|
|
|
Each catalog deployment has a number of _processors_ installed. They are
|
|
|
|
|
responsible for receiving unprocessed entities that the catalog decided are due
|
|
|
|
|
for processing, and then running that data through a number of processing
|
|
|
|
|
stages, mutating the entity and emitting auxiliary data about it. When all of
|
|
|
|
@@ -185,7 +185,28 @@ happen during ingestion or processing.
|
|
|
|
|
|
|
|
|
|
## Errors
|
|
|
|
|
|
|
|
|
|
> TODO: Describe how errors are exposed through entities
|
|
|
|
|
Errors during the ingestion and processing of entities can happen in a variety
|
|
|
|
|
of ways, and they may happen at a far later point in time than when they were
|
|
|
|
|
registered. For example, a registered file may get deleted in the remote system,
|
|
|
|
|
or the user may accidentally change the file contents in such a way that they
|
|
|
|
|
cannot be parsed successfully, etc.
|
|
|
|
|
|
|
|
|
|
There are two main ways that these errors are surfaced.
|
|
|
|
|
|
|
|
|
|
First, the catalog backend will produce detailed logs that should contain
|
|
|
|
|
sufficient information for a reader to find the causes for errors. Since these
|
|
|
|
|
logs are typically not easily found by end users, this can mainly be a useful
|
|
|
|
|
tool for Backstage operators who want to debug problems either with statically
|
|
|
|
|
registered entities that are under their control, or to help end users find
|
|
|
|
|
problems.
|
|
|
|
|
|
|
|
|
|
Second, for most classes of errors, the entity itself will contain a status
|
|
|
|
|
field that describes the problem. The contents of this field is shown at the top
|
|
|
|
|
of your entity page in Backstage, if you have placed the corresponding error
|
|
|
|
|
callout component (`EntityProcessingErrorsPanel`) there.
|
|
|
|
|
|
|
|
|
|
We are still working to improve the surfacing and observability around
|
|
|
|
|
processing loop errors.
|
|
|
|
|
|
|
|
|
|
## Orphaning
|
|
|
|
|
|
|
|
|
@@ -206,17 +227,22 @@ either, it becomes _orphaned_. The end result is as follows:
|
|
|
|
|
- The catalog page in Backstage for the child entity detects the new annotation
|
|
|
|
|
and informs users about the orphan status.
|
|
|
|
|
|
|
|
|
|
Orphaning can occur in several different scenarios. One common cause is that the
|
|
|
|
|
end user edited a corresponding catalog catalog-info YAML file removing the
|
|
|
|
|
entity's entry. In the case of a `Location` parent entity, orphaning can happen
|
|
|
|
|
if removing the target line pointing to the file containing the child entity.
|
|
|
|
|
Another common cause is large batch processors such as the ones that crawl
|
|
|
|
|
through remote systems looking for entities, no longer finding something that it
|
|
|
|
|
used to find before. Maybe the data was moved, or deleted, in the remote system.
|
|
|
|
|
So for example when a person leaves the company an LDAP org discovery processor
|
|
|
|
|
might leave an orphaned `User` entity behind. Note that this only applies to
|
|
|
|
|
processors - ingestion that happens using entity providers work differently,
|
|
|
|
|
described below.
|
|
|
|
|
Orphaning can occur in several different scenarios.
|
|
|
|
|
|
|
|
|
|
- If a catalog-info YAML file is moved from one place to another in the version
|
|
|
|
|
control system without updating the registration in the catalog, it will
|
|
|
|
|
effectively become orphaned "by" that registered location
|
|
|
|
|
- If the user edits a corresponding parent catalog-info YAML file removing the
|
|
|
|
|
entity's entry - for example in the case of a `Location` parent entity,
|
|
|
|
|
orphaning can happen if editing or removing the `target`/`targets` lines
|
|
|
|
|
pointing to the file containing the child entity.
|
|
|
|
|
- Another common cause is large batch processors such as the ones that crawl
|
|
|
|
|
through remote systems looking for entities, no longer finding something that
|
|
|
|
|
it used to find before. Maybe the data was moved, or deleted, in the remote
|
|
|
|
|
system. So for example when a person leaves the company an LDAP org discovery
|
|
|
|
|
processor might leave an orphaned `User` entity behind. Note that this only
|
|
|
|
|
applies to processors - ingestion that happens using entity providers work
|
|
|
|
|
differently, described below.
|
|
|
|
|
|
|
|
|
|
> Note that removing a file, or accidentally corrupting a file so that it cannot
|
|
|
|
|
> be read successfully, does _not_ lead to orphaning. Hard errors, including the
|
|
|
|
@@ -239,8 +265,78 @@ provided out of the box.
|
|
|
|
|
|
|
|
|
|
## Implicit Deletion
|
|
|
|
|
|
|
|
|
|
> TODO: Describe the process of entity providers eagerly deleting entities
|
|
|
|
|
Entity providers - not processors - are subject to _eager_ deletion of entities,
|
|
|
|
|
which may trigger the implicit deletion of more than just the entity you thought
|
|
|
|
|
you were deleting. This concept is explained here.
|
|
|
|
|
|
|
|
|
|
Recall that all entity providers manage a private "bucket" of entities, as
|
|
|
|
|
described in the [External integrations](external-integrations.md) article. They
|
|
|
|
|
can perform some operations on those entities, including additions, updates, and
|
|
|
|
|
deletions. Entity additions/updates are subject to the regular processing loops,
|
|
|
|
|
which means that bucket entities may end up forming roots of an entire graph of
|
|
|
|
|
entities that are emitted by those processors as they recursively work they way
|
|
|
|
|
through the bucket contents and its descendants.
|
|
|
|
|
|
|
|
|
|
When a provider issues a deletion of an entity in its bucket, that entity as
|
|
|
|
|
well as _the entire tree of entities processed out of it_, if any, are
|
|
|
|
|
considered for immediate deletion. Note "considered" - they are deleted if and
|
|
|
|
|
only if they would otherwise have become orphaned (no other parent entities
|
|
|
|
|
emitting them). Since the graph of entities is not strictly a tree, multiple
|
|
|
|
|
roots may actually end up indirectly referencing a node farther down in the
|
|
|
|
|
graph. If that's the case, that node won't go away until all such roots go away.
|
|
|
|
|
|
|
|
|
|
URLs to yaml files that you register using either the Create button or add to
|
|
|
|
|
your app-config, are both handled by entity providers. That means that this
|
|
|
|
|
implicit deletion mechanism comes into play in some everyday circumstances.
|
|
|
|
|
Let's illustrate.
|
|
|
|
|
|
|
|
|
|
Imagine that you have a monorepo, with a single `Location` entity in a
|
|
|
|
|
catalog-info file at the root, and that entity points to three other
|
|
|
|
|
catalog-info files in the repo with a `Component` entity in each one.
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
/
|
|
|
|
|
feature_one/
|
|
|
|
|
catalog-info.yaml <- kind: Component
|
|
|
|
|
feature_two/
|
|
|
|
|
catalog-info.yaml <- kind: Component
|
|
|
|
|
feature_three/
|
|
|
|
|
catalog-info.yaml <- kind: Component
|
|
|
|
|
catalog-info.yaml <- kind: Location
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
If you register the root `Location` entity, the actual effect is that _five_
|
|
|
|
|
entities appear in the catalog. First, one that is named `generated-`-something,
|
|
|
|
|
which corresponds to the registered URL itself. That's the one that the provider
|
|
|
|
|
has put in its "bucket". Then, as processing loops chug along, the `Location`
|
|
|
|
|
entity you pointed to appears as a child of that, and then the three `Component`
|
|
|
|
|
entities appear in turn as children of the `Location`.
|
|
|
|
|
|
|
|
|
|
As an end user of the Backstage interface, you may now want to delete one of the
|
|
|
|
|
three `Component` entities. You do that by visiting the three-dots menu in the
|
|
|
|
|
top right of an entity view. The popup dialog that appears will inform you that
|
|
|
|
|
actually this entity belongs to a certain root, and that you may want to remove
|
|
|
|
|
that root instead (which corresponds to unregistering the originally registered
|
|
|
|
|
URL). If you choose to do so, _all_ of the aforementioned five entities will
|
|
|
|
|
actually be deleted in the same operation.
|
|
|
|
|
|
|
|
|
|
If you did not want to perform this aggressive pruning, you might have instead
|
|
|
|
|
chosen to remove one of the `target` rows of your `Location` catalog-info file,
|
|
|
|
|
and then deleted the catalog-info file that contained the `Component` you wanted
|
|
|
|
|
to get rid of. Now the catalog would be left with an orphaned component, and you
|
|
|
|
|
would instead be able to use the explicit deletion (see below) to delete that
|
|
|
|
|
single component.
|
|
|
|
|
|
|
|
|
|
## Explicit Deletion
|
|
|
|
|
|
|
|
|
|
> TODO: Describe direct deletion via the catalog API
|
|
|
|
|
The catalog and its REST API also permits direct deletion of individual
|
|
|
|
|
entities. This makes sense to do on orphaned entities; entities that aren't
|
|
|
|
|
being actively kept up to date by any parent entities. The popup interface under
|
|
|
|
|
the three-dots menu option of entity views does offer this option, and the
|
|
|
|
|
orphaned status can be seen in an info box at the top of the entity's overview
|
|
|
|
|
page.
|
|
|
|
|
|
|
|
|
|
However, if you were to try to do an explicit depletion on an entity that's
|
|
|
|
|
being kept actively updated by a parent entity, it would just reappear again
|
|
|
|
|
shortly thereafter when the processing loops reconsider the parent entity that's
|
|
|
|
|
still in there.
|
|
|
|
|
Fredrik Adelöw