Merge branch 'master' of github.com:backstage/backstage into seant-splunk/awsS3_readTree_processor
This commit is contained in:
Sean Tan
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 17 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 22 KiB |
@@ -547,8 +547,8 @@ The current set of well-known and common values for this field is:
|
||||
|
||||
### `spec.owner` [required]
|
||||
|
||||
An [entity reference](#string-references) to the owner of the component, e.g.
|
||||
`artist-relations-team`. This field is required.
|
||||
An [entity reference](references.md#string-references) to the owner of the
|
||||
component, e.g. `artist-relations-team`. This field is required.
|
||||
|
||||
In Backstage, the owner of a component is the singular entity (commonly a team)
|
||||
that bears ultimate responsibility for the component, and has the authority and
|
||||
@@ -566,8 +566,8 @@ component, but there will always be one ultimate owner.
|
||||
|
||||
### `spec.system` [optional]
|
||||
|
||||
An [entity reference](#string-references) to the system that the component
|
||||
belongs to, e.g. `artist-engagement-portal`. This field is optional.
|
||||
An [entity reference](references.md#string-references) to the system that the
|
||||
component belongs to, e.g. `artist-engagement-portal`. This field is optional.
|
||||
|
||||
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
|
||||
| --------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------- |
|
||||
@@ -575,8 +575,8 @@ belongs to, e.g. `artist-engagement-portal`. This field is optional.
|
||||
|
||||
### `spec.subcomponentOf` [optional]
|
||||
|
||||
An [entity reference](#string-references) to another component of which the
|
||||
component is a part, e.g. `spotify-ios-app`. This field is optional.
|
||||
An [entity reference](references.md#string-references) to another component of
|
||||
which the component is a part, e.g. `spotify-ios-app`. This field is optional.
|
||||
|
||||
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
|
||||
| ---------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------- |
|
||||
@@ -584,8 +584,8 @@ component is a part, e.g. `spotify-ios-app`. This field is optional.
|
||||
|
||||
### `spec.providesApis` [optional]
|
||||
|
||||
An array of [entity references](#string-references) to the APIs that are
|
||||
provided by the component, e.g. `artist-api`. This field is optional.
|
||||
An array of [entity references](references.md#string-references) to the APIs
|
||||
that are provided by the component, e.g. `artist-api`. This field is optional.
|
||||
|
||||
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
|
||||
| --------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------------- |
|
||||
@@ -593,8 +593,8 @@ provided by the component, e.g. `artist-api`. This field is optional.
|
||||
|
||||
### `spec.consumesApis` [optional]
|
||||
|
||||
An array of [entity references](#string-references) to the APIs that are
|
||||
consumed by the component, e.g. `artist-api`. This field is optional.
|
||||
An array of [entity references](references.md#string-references) to the APIs
|
||||
that are consumed by the component, e.g. `artist-api`. This field is optional.
|
||||
|
||||
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
|
||||
| --------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------------- |
|
||||
@@ -602,9 +602,9 @@ consumed by the component, e.g. `artist-api`. This field is optional.
|
||||
|
||||
### `spec.dependsOn` [optional]
|
||||
|
||||
An array of [entity references](#string-references) to the components and
|
||||
resources that the component depends on, e.g. `artists-db`. This field is
|
||||
optional.
|
||||
An array of [entity references](references.md#string-references) to the
|
||||
components and resources that the component depends on, e.g. `artists-db`. This
|
||||
field is optional.
|
||||
|
||||
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
|
||||
| --------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------- |
|
||||
@@ -732,8 +732,8 @@ You can find out more about the `steps` key
|
||||
|
||||
### `spec.owner` [optional]
|
||||
|
||||
An [entity reference](#string-references) to the owner of the template, e.g.
|
||||
`artist-relations-team`. This field is required.
|
||||
An [entity reference](references.md#string-references) to the owner of the
|
||||
template, e.g. `artist-relations-team`. This field is required.
|
||||
|
||||
In Backstage, the owner of a Template is the singular entity (commonly a team)
|
||||
that bears ultimate responsibility for the Template, and has the authority and
|
||||
@@ -842,8 +842,8 @@ The current set of well-known and common values for this field is:
|
||||
|
||||
### `spec.owner` [required]
|
||||
|
||||
An [entity reference](#string-references) to the owner of the component, e.g.
|
||||
`artist-relations-team`. This field is required.
|
||||
An [entity reference](references.md#string-references) to the owner of the
|
||||
component, e.g. `artist-relations-team`. This field is required.
|
||||
|
||||
In Backstage, the owner of an API is the singular entity (commonly a team) that
|
||||
bears ultimate responsibility for the API, and has the authority and capability
|
||||
@@ -861,8 +861,8 @@ one ultimate owner.
|
||||
|
||||
### `spec.system` [optional]
|
||||
|
||||
An [entity reference](#string-references) to the system that the API belongs to,
|
||||
e.g. `artist-engagement-portal`. This field is optional.
|
||||
An [entity reference](references.md#string-references) to the system that the
|
||||
API belongs to, e.g. `artist-engagement-portal`. This field is optional.
|
||||
|
||||
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
|
||||
| --------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------- |
|
||||
@@ -1069,8 +1069,8 @@ Exactly equal to `backstage.io/v1alpha1` and `Resource`, respectively.
|
||||
|
||||
### `spec.owner` [required]
|
||||
|
||||
An [entity reference](#string-references) to the owner of the resource, e.g.
|
||||
`artist-relations-team`. This field is required.
|
||||
An [entity reference](references.md#string-references) to the owner of the
|
||||
resource, e.g. `artist-relations-team`. This field is required.
|
||||
|
||||
In Backstage, the owner of a resource is the singular entity (commonly a team)
|
||||
that bears ultimate responsibility for the resource, and has the authority and
|
||||
@@ -1101,8 +1101,8 @@ Some common values for this field could be:
|
||||
|
||||
### `spec.system` [optional]
|
||||
|
||||
An [entity reference](#string-references) to the system that the resource
|
||||
belongs to, e.g. `artist-engagement-portal`. This field is optional.
|
||||
An [entity reference](references.md#string-references) to the system that the
|
||||
resource belongs to, e.g. `artist-engagement-portal`. This field is optional.
|
||||
|
||||
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
|
||||
| --------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------- |
|
||||
@@ -1110,9 +1110,9 @@ belongs to, e.g. `artist-engagement-portal`. This field is optional.
|
||||
|
||||
### `spec.dependsOn` [optional]
|
||||
|
||||
An array of [entity references](#string-references) to the components and
|
||||
resources that the resource depends on, e.g. `artist-lookup`. This field is
|
||||
optional.
|
||||
An array of [entity references](references.md#string-references) to the
|
||||
components and resources that the resource depends on, e.g. `artist-lookup`.
|
||||
This field is optional.
|
||||
|
||||
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
|
||||
| --------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------- |
|
||||
@@ -1156,8 +1156,8 @@ Exactly equal to `backstage.io/v1alpha1` and `System`, respectively.
|
||||
|
||||
### `spec.owner` [required]
|
||||
|
||||
An [entity reference](#string-references) to the owner of the system, e.g.
|
||||
`artist-relations-team`. This field is required.
|
||||
An [entity reference](references.md#string-references) to the owner of the
|
||||
system, e.g. `artist-relations-team`. This field is required.
|
||||
|
||||
In Backstage, the owner of a system is the singular entity (commonly a team)
|
||||
that bears ultimate responsibility for the system, and has the authority and
|
||||
@@ -1175,8 +1175,8 @@ but there will always be one ultimate owner.
|
||||
|
||||
### `spec.domain` [optional]
|
||||
|
||||
An [entity reference](#string-references) to the domain that the system belongs
|
||||
to, e.g. `artists`. This field is optional.
|
||||
An [entity reference](references.md#string-references) to the domain that the
|
||||
system belongs to, e.g. `artists`. This field is optional.
|
||||
|
||||
| [`kind`](#apiversion-and-kind-required) | Default [`namespace`](#namespace-optional) | Generated [relation](well-known-relations.md) type |
|
||||
| --------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------- |
|
||||
@@ -1215,8 +1215,8 @@ Exactly equal to `backstage.io/v1alpha1` and `Domain`, respectively.
|
||||
|
||||
### `spec.owner` [required]
|
||||
|
||||
An [entity reference](#string-references) to the owner of the domain, e.g.
|
||||
`artist-relations-team`. This field is required.
|
||||
An [entity reference](references.md#string-references) to the owner of the
|
||||
domain, e.g. `artist-relations-team`. This field is required.
|
||||
|
||||
In Backstage, the owner of a domain is the singular entity (commonly a team)
|
||||
that bears ultimate responsibility for the domain, and has the authority and
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
id: life-of-an-entity
|
||||
title: The Life of an Entity
|
||||
sidebar_label: The Life of an Entity
|
||||
# prettier-ignore
|
||||
description: The life cycle of entities, from being introduced into the catalog, through processing, to being removed again
|
||||
---
|
||||
|
||||
This document gives a high level overview of the catalog backend, and the
|
||||
technical processes involved in making entities flow through it. It is mainly
|
||||
aimed at developers who want to understand the internals while installing or
|
||||
extending the catalog. However, it can be informative for other personas too.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
The catalog forms a hub of sorts, where entities are ingested from various
|
||||
authoritative sources and held in a database, subject to automated processing,
|
||||
and then presented through an API for quick and easy access by Backstage and
|
||||
others. The most common source is [YAML files](descriptor-format.md) on a
|
||||
standard format, living in version control systems near the source code of
|
||||
systems that they describe. Those files are registered with the catalog and
|
||||
maintained by the respective owners. The catalog makes sure to keep itself up to
|
||||
date with changes to those files.
|
||||
|
||||
The main extension points where developers can customize the catalog are:
|
||||
|
||||
- _Entity providers_, that feed initial raw entity data into the catalog,
|
||||
- _Policies_, that establish baseline rules about the shape of entities,
|
||||
- _Processors_, that validate, analyze, and mutate the raw entity data into its
|
||||
final form.
|
||||
|
||||
The high level processes involved are:
|
||||
|
||||
- _Ingestion_, where entity providers fetch raw entity data from external
|
||||
sources and seed it into the database,
|
||||
- _Processing_, where the policies and processors continually treat the ingested
|
||||
data and may emit both other raw entities (that are also subject to
|
||||
processing), errors, relations to other entities, etc.,
|
||||
- _Stitching_, where all of the data emitted by various processors are assembled
|
||||
together into the final output entity.
|
||||
|
||||
An entity is not visible to the outside world (through the catalog API), until
|
||||
it has passed through the last process and landed among the final entities.
|
||||
|
||||
|
||||
|
||||
The details of these processes are described below.
|
||||
|
||||
## Ingestion
|
||||
|
||||
Each catalog deployment has a number of entity providers installed. They are
|
||||
responsible for fetching data from external authoritative sources in any way
|
||||
that they see fit, to translate those into entity objects, and to notify the
|
||||
database when those entities are added or removed. These are the _unprocessed
|
||||
entities_ that will be subject to later processing (see below), and they form
|
||||
the very basis of existence for entities. If there were no entity providers, no
|
||||
entities would ever enter the system.
|
||||
|
||||
The database always keeps track of the set of entities that belong to each
|
||||
provider; no two providers can try to output the same entity. And when a
|
||||
provider signals the removal of an entity, then that leads to an _eager
|
||||
deletion_: the entity and all auxiliary data that it has led to in the database
|
||||
is immediately purged.
|
||||
|
||||
|
||||
|
||||
There are two providers installed by default: the one that deals with user
|
||||
registered locations (e.g. URLs to YAML files), and the one that deals with
|
||||
static locations in the app-config. You can add more third party providers by
|
||||
passing them to the catalog builder in your backend initialization code, and you
|
||||
can easily write your own.
|
||||
|
||||
An entity provider is a class that implements the `EntityProvider` interface. It
|
||||
has three main parts:
|
||||
|
||||
- The identity: Each provider instance has a unique, stable identifier that the
|
||||
database can use to keep track of the originator of each unprocessed entity.
|
||||
- The connection: During backend startup, each provider is attached to the
|
||||
catalog runtime.
|
||||
- The stream of events: During its lifetime, the provider can issue change
|
||||
events to the runtime at any point in time, to modify its set of unprocessed
|
||||
entities.
|
||||
|
||||
It is entirely up to the provider to choose how and when it produces these
|
||||
change events. For example, the app-config provider only fires off an update at
|
||||
startup and then lies dormant. The location database provider does an initial
|
||||
update at startup, and then small delta updates every time a location database
|
||||
change is detected. The LDAP provider is driven externally by a timer loop that
|
||||
occasionally triggers a full update. Some future provider may be entirely event
|
||||
driven, feeding off an event bus or web hook. There is no magic coordination
|
||||
among providers; if they need to arrange synchronization or locking among
|
||||
themselves for example to avoid duplicate work across multiple catalog service
|
||||
machines, they need to handle that out-of-band.
|
||||
|
||||
The entities that are emitted get some coarse validation applied to them, to
|
||||
ensure that they at least adhere to the most basic schema rules about how an
|
||||
entity should be shaped. For example, they need to have a `kind`, a
|
||||
`metadata.name`, and optionally a `metadata.namespace`, among others. Apart from
|
||||
that, the ingestion stage considers its work done, and stores the unprocessed
|
||||
entities to be picked up at a later time by the processing system. This means
|
||||
that the more precise validation rules that you put in place on entities are
|
||||
_not_ yet applied at this stage.
|
||||
|
||||
## Processing
|
||||
|
||||
Every unprocessed entity comes with a timestamp, which tells at what time that
|
||||
the processing loop should next try to process it. When the entity first
|
||||
appears, this timestamp is set to "now" - asking for it to be picked up as soon
|
||||
as possible.
|
||||
|
||||
Each catalog deployment has a number of processors installed. They are
|
||||
responsible for receiving unprocessed entities that the catalog decided are due
|
||||
for processing, and then running that data through a number of processing
|
||||
stages. mutating the entity and emitting auxiliary data about it. When all of
|
||||
that is done, the catalog takes all of that information and stores it as the
|
||||
processed entity, and errors and relations to other entities separately. Then,
|
||||
the catalog checks to see what entities are touched by that output, and triggers
|
||||
the final assembly of those (see Stitching below).
|
||||
|
||||
There are several stages involved in the processing.
|
||||
|
||||
> TODO: More info here
|
||||
|
||||
## Stitching
|
||||
|
||||
The stitching is currently a fixed process, that cannot be modified or extended.
|
||||
This means that any modifications you want to make on the final result, has to
|
||||
happen during ingestion or processing.
|
||||
|
||||
> TODO: More info here
|
||||
@@ -148,12 +148,18 @@ const AppRoutes = () => {
|
||||
> an external storage provider and external CI/CD is used). If you use the
|
||||
> "basic" or "out-of-the-box" setup, you can stop here! No action needed.
|
||||
|
||||
The beta version of TechDocs (v0.x.y) made a breaking change to the way TechDocs
|
||||
content was accessed and stored, allowing pages to be accessed with
|
||||
case-insensitive entity triplet paths (e.g. `/docs/namespace/kind/name` whereas
|
||||
in prior versions, they could only be accessed at `/docs/namespace/Kind/name`).
|
||||
In order to enable this change, documentation has to be stored in an external
|
||||
storage provider using an object key whose entity triplet is lower-cased.
|
||||
For the purposes of this guide, TechDocs Beta version is defined as:
|
||||
|
||||
- **TechDocs Plugin**: At least `v0.11.0`
|
||||
- **TechDocs Backend Plugin**: At least `v0.10.0`
|
||||
- **TechDocs CLI**: At least `v0.7.0`
|
||||
|
||||
The beta version of TechDocs made a breaking change to the way TechDocs content
|
||||
was accessed and stored, allowing pages to be accessed with case-insensitive
|
||||
entity triplet paths (e.g. `/docs/namespace/kind/name` whereas in prior
|
||||
versions, they could only be accessed at `/docs/namespace/Kind/name`). In order
|
||||
to enable this change, documentation has to be stored in an external storage
|
||||
provider using an object key whose entity triplet is lower-cased.
|
||||
|
||||
New installations of TechDocs since the beta version will work fine with no
|
||||
action, but for those who were running TechDocs prior to this version, a
|
||||
|
||||
@@ -128,7 +128,7 @@ const themeOptions = createThemeOptions({
|
||||
tool: genPageTheme(['#123456','#123456'], shapes.round),
|
||||
service: genPageTheme(['#123456','#123456'], shapes.wave),
|
||||
website: genPageTheme(['#123456','#123456'], shapes.wave),
|
||||
library: genPageTheme(['#123456','#123456'] shapes.wave),
|
||||
library: genPageTheme(['#123456','#123456'], shapes.wave),
|
||||
other: genPageTheme(['#123456','#123456'], shapes.wave),
|
||||
app: genPageTheme(['#123456','#123456'], shapes.wave),
|
||||
apis: genPageTheme(['#123456','#123456'], shapes.wave),
|
||||
|
||||
Reference in New Issue
Block a user