paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 40 Packages Projects Releases Wiki Activity

Merge branch 'backstage:master' into feature/k8s-dashboard-links

Browse Source
This commit is contained in:
livetocode
2021-09-11 18:25:34 -04:00
committed by GitHub
parent 8fcdfaf1af 7f221c6c6b
commit 07a8543ffa
589 changed files with 9535 additions and 5045 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/features/software-catalog/catalog-customization.md
+8 -9
View File
@@ -34,15 +34,14 @@ export const CustomCatalogPage = ({
}: CatalogPageProps) => {
return (
<PageWithHeader title={`${orgName} Catalog`} themeId="home">
<Content>
<ContentHeader title="Components">
<CreateButton title="Create Component" to={link} />
<SupportButton>All your software catalog entities</SupportButton>
</ContentHeader>
<EntityListProvider>
<EntityListProvider>
<Content>
<ContentHeader titleComponent={<CatalogKindHeader />}>
<CreateButton title="Create Component" to={link} />
<SupportButton>All your software catalog entities</SupportButton>
</ContentHeader>
<FilteredEntityLayout>
<FilterContainer>
<EntityKindPicker initialFilter="component" hidden />
<EntityTypePicker />
<UserListPicker initialFilter={initiallySelectedFilter} />
<EntityTagPicker />
@@ -51,8 +50,8 @@ export const CustomCatalogPage = ({
<CatalogTable columns={columns} actions={actions} />
</EntityListContainer>
</FilteredEntityLayout>
</EntityListProvider>
</Content>
</Content>
</EntityListProvider>
</PageWithHeader>
);
};
docs/features/software-catalog/descriptor-format.md
+48 -38
View File
@@ -245,6 +245,22 @@ the entity belongs to the `"default"` namespace.
Namespaces may also be part of the catalog, and are `v1` / `Namespace` entities,
i.e. not Backstage specific but the same as in Kubernetes.
### `title` [optional]
A display name of the entity, to be presented in user interfaces instead of the
`name` property above, when available.
This field is sometimes useful when the `name` is cumbersome or ends up being
perceived as overly technical. The title generally does not have as stringent
format requirements on it, so it may contain special characters and be more
explanatory. Do keep it very short though, and avoid situations where a title
can be confused with the name of another entity, or where two entities share a
title.
Note that this is only for display purposes, and may be ignored by some parts of
the code. [Entity references](references.md) still always make use of the `name`
property for example, not the title.
### `description` [optional]
A human readable description of the entity, to be shown in Backstage. Should be
@@ -531,8 +547,8 @@ The current set of well-known and common values for this field is:
### `spec.owner` [required]
An [entity reference](#string-references) to the owner of the component, e.g.
`artist-relations-team`. This field is required.
An [entity reference](references.md#string-references) to the owner of the
component, e.g. `artist-relations-team`. This field is required.
In Backstage, the owner of a component is the singular entity (commonly a team)
that bears ultimate responsibility for the component, and has the authority and
@@ -550,8 +566,8 @@ component, but there will always be one ultimate owner.
### `spec.system` [optional]
An [entity reference](#string-references) to the system that the component
belongs to, e.g. `artist-engagement-portal`. This field is optional.
An [entity reference](references.md#string-references) to the system that the
component belongs to, e.g. `artist-engagement-portal`. This field is optional.
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
| --------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------- |
@@ -559,8 +575,8 @@ belongs to, e.g. `artist-engagement-portal`. This field is optional.
### `spec.subcomponentOf` [optional]
An [entity reference](#string-references) to another component of which the
component is a part, e.g. `spotify-ios-app`. This field is optional.
An [entity reference](references.md#string-references) to another component of
which the component is a part, e.g. `spotify-ios-app`. This field is optional.
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
| ---------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------- |
@@ -568,8 +584,8 @@ component is a part, e.g. `spotify-ios-app`. This field is optional.
### `spec.providesApis` [optional]
An array of [entity references](#string-references) to the APIs that are
provided by the component, e.g. `artist-api`. This field is optional.
An array of [entity references](references.md#string-references) to the APIs
that are provided by the component, e.g. `artist-api`. This field is optional.
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
| --------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------------- |
@@ -577,8 +593,8 @@ provided by the component, e.g. `artist-api`. This field is optional.
### `spec.consumesApis` [optional]
An array of [entity references](#string-references) to the APIs that are
consumed by the component, e.g. `artist-api`. This field is optional.
An array of [entity references](references.md#string-references) to the APIs
that are consumed by the component, e.g. `artist-api`. This field is optional.
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
| --------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------------- |
@@ -586,9 +602,9 @@ consumed by the component, e.g. `artist-api`. This field is optional.
### `spec.dependsOn` [optional]
An array of [entity references](#string-references) to the components and
resources that the component depends on, e.g. `artists-db`. This field is
optional.
An array of [entity references](references.md#string-references) to the
components and resources that the component depends on, e.g. `artists-db`. This
field is optional.
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
| --------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------- |
@@ -690,12 +706,6 @@ shape, this kind has the following structure.
Exactly equal to `backstage.io/v1beta2` and `Template`, respectively.
### `metadata.title` [required]
The nice display name for the template as a string, e.g. `React SSR Template`.
This field is required as is used to reference the template to the user instead
of the `metadata.name` field.
### `metadata.tags` [optional]
A list of strings that can be associated with the template, e.g.
@@ -722,8 +732,8 @@ You can find out more about the `steps` key
### `spec.owner` [optional]
An [entity reference](#string-references) to the owner of the template, e.g.
`artist-relations-team`. This field is required.
An [entity reference](references.md#string-references) to the owner of the
template, e.g. `artist-relations-team`. This field is required.
In Backstage, the owner of a Template is the singular entity (commonly a team)
that bears ultimate responsibility for the Template, and has the authority and
@@ -832,8 +842,8 @@ The current set of well-known and common values for this field is:
### `spec.owner` [required]
An [entity reference](#string-references) to the owner of the component, e.g.
`artist-relations-team`. This field is required.
An [entity reference](references.md#string-references) to the owner of the
component, e.g. `artist-relations-team`. This field is required.
In Backstage, the owner of an API is the singular entity (commonly a team) that
bears ultimate responsibility for the API, and has the authority and capability
@@ -851,8 +861,8 @@ one ultimate owner.
### `spec.system` [optional]
An [entity reference](#string-references) to the system that the API belongs to,
e.g. `artist-engagement-portal`. This field is optional.
An [entity reference](references.md#string-references) to the system that the
API belongs to, e.g. `artist-engagement-portal`. This field is optional.
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
| --------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------- |
@@ -1059,8 +1069,8 @@ Exactly equal to `backstage.io/v1alpha1` and `Resource`, respectively.
### `spec.owner` [required]
An [entity reference](#string-references) to the owner of the resource, e.g.
`artist-relations-team`. This field is required.
An [entity reference](references.md#string-references) to the owner of the
resource, e.g. `artist-relations-team`. This field is required.
In Backstage, the owner of a resource is the singular entity (commonly a team)
that bears ultimate responsibility for the resource, and has the authority and
@@ -1091,8 +1101,8 @@ Some common values for this field could be:
### `spec.system` [optional]
An [entity reference](#string-references) to the system that the resource
belongs to, e.g. `artist-engagement-portal`. This field is optional.
An [entity reference](references.md#string-references) to the system that the
resource belongs to, e.g. `artist-engagement-portal`. This field is optional.
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
| --------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------- |
@@ -1100,9 +1110,9 @@ belongs to, e.g. `artist-engagement-portal`. This field is optional.
### `spec.dependsOn` [optional]
An array of [entity references](#string-references) to the components and
resources that the resource depends on, e.g. `artist-lookup`. This field is
optional.
An array of [entity references](references.md#string-references) to the
components and resources that the resource depends on, e.g. `artist-lookup`.
This field is optional.
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
| --------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------- |
@@ -1146,8 +1156,8 @@ Exactly equal to `backstage.io/v1alpha1` and `System`, respectively.
### `spec.owner` [required]
An [entity reference](#string-references) to the owner of the system, e.g.
`artist-relations-team`. This field is required.
An [entity reference](references.md#string-references) to the owner of the
system, e.g. `artist-relations-team`. This field is required.
In Backstage, the owner of a system is the singular entity (commonly a team)
that bears ultimate responsibility for the system, and has the authority and
@@ -1165,8 +1175,8 @@ but there will always be one ultimate owner.
### `spec.domain` [optional]
An [entity reference](#string-references) to the domain that the system belongs
to, e.g. `artists`. This field is optional.
An [entity reference](references.md#string-references) to the domain that the
system belongs to, e.g. `artists`. This field is optional.
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
| --------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------- |
@@ -1205,8 +1215,8 @@ Exactly equal to `backstage.io/v1alpha1` and `Domain`, respectively.
### `spec.owner` [required]
An [entity reference](#string-references) to the owner of the domain, e.g.
`artist-relations-team`. This field is required.
An [entity reference](references.md#string-references) to the owner of the
domain, e.g. `artist-relations-team`. This field is required.
In Backstage, the owner of a domain is the singular entity (commonly a team)
that bears ultimate responsibility for the domain, and has the authority and
docs/features/software-catalog/life-of-an-entity.md
+246
View File
@@ -0,0 +1,246 @@
---
id: life-of-an-entity
title: The Life of an Entity
sidebar_label: The Life of an Entity
# prettier-ignore
description: The life cycle of entities, from being introduced into the catalog, through processing, to being removed again
---
This document gives a high level overview of the catalog backend, and the
technical processes involved in making entities flow through it. It is mainly
aimed at developers who want to understand the internals while installing or
extending the catalog. However, it can be informative for other personas too.
## Key Concepts
The catalog forms a hub of sorts, where entities are ingested from various
authoritative sources and held in a database, subject to automated processing,
and then presented through an API for quick and easy access by Backstage and
others. The most common source is [YAML files](descriptor-format.md) on a
standard format, living in version control systems near the source code of
systems that they describe. Those files are registered with the catalog and
maintained by the respective owners. The catalog makes sure to keep itself up to
date with changes to those files.
The main extension points where developers can customize the catalog are:
- _Entity providers_, that feed initial raw entity data into the catalog,
- _Policies_, that establish baseline rules about the shape of entities,
- _Processors_, that validate, analyze, and mutate the raw entity data into its
final form.
The high level processes involved are:
- _Ingestion_, where entity providers fetch raw entity data from external
sources and seed it into the database,
- _Processing_, where the policies and processors continually treat the ingested
data and may emit both other raw entities (that are also subject to
processing), errors, relations to other entities, etc.,
- _Stitching_, where all of the data emitted by various processors are assembled
together into the final output entity.
An entity is not visible to the outside world (through the catalog API), until
it has passed through the last process and landed among the final entities.
![General overview](../../assets/features/catalog/life-of-an-entity_overview.svg)
The details of these processes are described below.
## Ingestion
Each catalog deployment has a number of entity providers installed. They are
responsible for fetching data from external authoritative sources in any way
that they see fit, to translate those into entity objects, and to notify the
database when those entities are added or removed. These are the _unprocessed
entities_ that will be subject to later processing (see below), and they form
the very basis of existence for entities. If there were no entity providers, no
entities would ever enter the system.
The database always keeps track of the set of entities that belong to each
provider; no two providers can try to output the same entity. And when a
provider signals the removal of an entity, then that leads to an _eager
deletion_: the entity and all auxiliary data that it has led to in the database
is immediately purged.
![Ingestion overview](../../assets/features/catalog/life-of-an-entity_ingestion.svg)
There are two providers installed by default: the one that deals with user
registered locations (e.g. URLs to YAML files), and the one that deals with
static locations in the app-config. You can add more third party providers by
passing them to the catalog builder in your backend initialization code, and you
can easily write your own.
An entity provider is a class that implements the `EntityProvider` interface. It
has three main parts:
- The identity: Each provider instance has a unique, stable identifier that the
database can use to keep track of the originator of each unprocessed entity.
- The connection: During backend startup, each provider is attached to the
catalog runtime.
- The stream of events: During its lifetime, the provider can issue change
events to the runtime at any point in time, to modify its set of unprocessed
entities.
It is entirely up to the provider to choose how and when it produces these
change events. For example, the app-config provider only fires off an update at
startup and then lies dormant. The location database provider does an initial
update at startup, and then small delta updates every time a location database
change is detected. The LDAP provider is driven externally by a timer loop that
occasionally triggers a full update. Some future provider may be entirely event
driven, feeding off an event bus or web hook. There is no magic coordination
among providers; if they need to arrange synchronization or locking among
themselves for example to avoid duplicate work across multiple catalog service
machines, they need to handle that out-of-band.
The entities that are emitted get some coarse validation applied to them, to
ensure that they at least adhere to the most basic schema rules about how an
entity should be shaped. For example, they need to have a `kind`, a
`metadata.name`, and optionally a `metadata.namespace`, among others. Apart from
that, the ingestion stage considers its work done, and stores the unprocessed
entities to be picked up at a later time by the processing system. This means
that the more precise validation rules that you put in place on entities are
_not_ yet applied at this stage.
## Processing
Every unprocessed entity comes with a timestamp, which tells at what time that
the processing loop should next try to process it. When the entity first
appears, this timestamp is set to "now" - asking for it to be picked up as soon
as possible.
Each catalog deployment has a number of processors installed. They are
responsible for receiving unprocessed entities that the catalog decided are due
for processing, and then running that data through a number of processing
stages, mutating the entity and emitting auxiliary data about it. When all of
that is done, the catalog takes all of that information and stores it as the
processed entity, and errors and relations to other entities separately. Then,
the catalog checks to see what entities are touched by that output, and triggers
the final assembly of those (see Stitching below).
![Processing overview](../../assets/features/catalog/life-of-an-entity_processing.svg)
Entities are always processed one by one, but all of your catalog service hosts
collaborate in doing so to distribute the load. Note how each processor can
contribute to one or more of the fixed steps in the processing pipeline. First
all of the processors' contributions to one step are run in the order that the
processors were registered, then all of their contributions to the next step in
the same order, and so on.
Each step has the opportunity to optionally modify the entity, and to optionally
emit other information. For example, the processor might look at information in
the `spec` field of the entity, and emit relations that correspond to those
declarations. If the processor emits an entity, then that entity gets stored
verbatim with a timestamp saying that it, too, should be processed as soon as
possible. If errors are emitted, then that signals that something is wrong with
the entity and that it should not replace whatever previously error-free version
we had among the final entities. If relations are emitted, then they are put in
a dedicated relations table to be picked up by the stitching process below.
> Optional low level detail note: When entities are emitted, the catalog keeps
> track of the edges between the emitting entity and the ones emitted. This
> happens behind the scenes, hidden from the outside, and is used to form a
> graph. This is _not_ the same thing as relations! The purpose of these edges,
> is to be able to detect when an entity becomes orphaned (see below), and to be
> able to perform eager deletions throughout the graph when a root is explicitly
> unregistered and nothing else is keeping lower nodes alive. We will talk more
> about orphaning and deletions later on in this article.
When the final step has completed, and no errors were encountered, the processed
entity and all of the relations are finally persisted in the database. Then the
catalog considers this entity, and all of the entities it had relations to,
subject for stitching.
It is worth noting here that the processing does not lead to deletion or
unregistration of entities; it can only call new entities into existence or
update entities that it has previously called into existence. More about that
later.
## Stitching
Stitching finalizes the entity, by gathering all of the output from the previous
steps and merging them into the final object which is what is visible from the
catalog API. As the final entity itself gets updated, the stitcher makes sure
that the search table gets refreshed accordingly as well.
> Note: The search table mentioned here is not related to the core Search
> feature of Backstage. It's rather the table that backs the ability to filter
> catalog API query results.
![Stitching overview](../../assets/features/catalog/life-of-an-entity_stitching.svg)
The diagram shows how the stitcher reads from several sources:
- The processed entity, as returned from the processing step
- The errors, if any, that were emitted by the processing step
- All relations that were emitted by the processing step, as well as any
relations emitted by _other_ entity processing steps that happen to point at
the current entity
The last part is noteworthy: This is how the stitcher is able to collect all of
the relation edges, both incoming and outgoing, no matter who produced them.
The stitching is currently a fixed process, that cannot be modified or extended.
This means that any modifications you want to make on the final result, has to
happen during ingestion or processing.
## Errors
> TODO: Describe how errors are exposed through entities
## Orphaning
As mentioned earlier, entities internally form a graph. The edges go from
processed parent entities, to child entities emitted while processing the
parent.
The processing loop runs continuously, so these edges are reconsidered over
time. If processing a parent entity no longer emits a given child entity, then
that former edge is severed. If that child has no other edges pointing at it
either, it becomes _orphaned_. The end result is as follows:
- The stitching process injects a `backstage.io/orphan: 'true'` annotation on
the child entity.
- The child entity is _not_ removed from the catalog, but stays around until
explicitly deleted via the catalog API, or "reclaimed" by the original parent
or another parent starting to reference it.
- The catalog page in Backstage for the child entity detects the new annotation
and informs users about the orphan status.
Orphaning can occur in several different scenarios. One common cause is that the
end user edited a corresponding catalog catalog-info YAML file removing the
entity's entry. In the case of a `Location` parent entity, orphaning can happen
if removing the target line pointing to the file containing the child entity.
Another common cause is large batch processors such as the ones that crawl
through remote systems looking for entities, no longer finding something that it
used to find before. Maybe the data was moved, or deleted, in the remote system.
So for example when a person leaves the company an LDAP org discovery processor
might leave an orphaned `User` entity behind. Note that this only applies to
processors - ingestion that happens using entity providers work differently,
described below.
> Note that removing a file, or accidentally corrupting a file so that it cannot
> be read successfully, does _not_ lead to orphaning. Hard errors, including the
> inability to find or read a distinct remote, are marked as such on the entity
> to inform the owner that something is wrong. But processing and other
> behaviors continue as usual.
The reason that the orphaning mechanism exists instead of having an eager
deletion triggered, is safety. Scenarios like these can happen purely by
accident, due to the asynchronous nature of the system and the fallible nature
of humans. In particular when external systems start consuming and relying on
the catalog, there could be substantial consequences to suddenly dropping
entities without explicit owner consent. The catalog therefore takes the stance
that entities that often were added by direct user action should also be deleted
only by direct user action.
It is possible to use the catalog API to build automated "reaper" systems that
finally delete entities that are orphaned. This is however not something that's
provided out of the box.
## Implicit Deletion
> TODO: Describe the process of entity providers eagerly deleting entities
## Explicit Deletion
> TODO: Describe direct deletion via the catalog API
docs/features/techdocs/README.md
+1 -1
View File
@@ -29,7 +29,7 @@ about TechDocs and the philosophy in its
- Explore and take advantage of the large ecosystem of
[MkDocs plugins](https://www.mkdocs.org/user-guide/plugins/) to create a rich
reading experience.
- Search for and find docs (coming soon).
- Search for and find docs.
- Highlight text and raise an Issue to create feedback loop to drive quality
documentation (future).
- Contribute to and deploy from a marketplace of TechDocs widgets (future).
docs/features/techdocs/how-to-guides.md
+12 -6
View File
@@ -148,12 +148,18 @@ const AppRoutes = () => {
> an external storage provider and external CI/CD is used). If you use the
> "basic" or "out-of-the-box" setup, you can stop here! No action needed.
The beta version of TechDocs (v0.x.y) made a breaking change to the way TechDocs
content was accessed and stored, allowing pages to be accessed with
case-insensitive entity triplet paths (e.g. `/docs/namespace/kind/name` whereas
in prior versions, they could only be accessed at `/docs/namespace/Kind/name`).
In order to enable this change, documentation has to be stored in an external
storage provider using an object key whose entity triplet is lower-cased.
For the purposes of this guide, TechDocs Beta version is defined as:
- **TechDocs Plugin**: At least `v0.11.0`
- **TechDocs Backend Plugin**: At least `v0.10.0`
- **TechDocs CLI**: At least `v0.7.0`
The beta version of TechDocs made a breaking change to the way TechDocs content
was accessed and stored, allowing pages to be accessed with case-insensitive
entity triplet paths (e.g. `/docs/namespace/kind/name` whereas in prior
versions, they could only be accessed at `/docs/namespace/Kind/name`). In order
to enable this change, documentation has to be stored in an external storage
provider using an object key whose entity triplet is lower-cased.
New installations of TechDocs since the beta version will work fine with no
action, but for those who were running TechDocs prior to this version, a