Rework catalog configuration; add integrations documentation
Signed-off-by: Tim Hansen <timbonicus@gmail.com>
This commit is contained in:
Tim Hansen
@@ -143,6 +143,7 @@ css
|
||||
dariddler
|
||||
dataflow
|
||||
deadnaming
|
||||
declaratively
|
||||
destructured
|
||||
dev
|
||||
devops
|
||||
|
||||
@@ -11,111 +11,13 @@ such as reading raw entity data from a remote source, parsing it, transforming
|
||||
it, and validating it. These processors are configured under the
|
||||
`catalog.processors` configuration key.
|
||||
|
||||
### Processor: url
|
||||
### Static Location Configuration
|
||||
|
||||
The `url` processor is responsible for fetching entity data from files in any
|
||||
external provider like GitHub, GitLab, Bitbucket, etc. Unlike other processors,
|
||||
the configuration of this processor lives under the top-level `integrations`
|
||||
key, as it is used by other parts of Backstage too.
|
||||
The simplest configuration for the catalog, as shown in the default
|
||||
`@backstage/create-app` template, is to declaratively add locations pointing to
|
||||
YAML files with [static configuration](../../conf/index.md).
|
||||
|
||||
```yaml
|
||||
integrations:
|
||||
github:
|
||||
- host: github.com
|
||||
token:
|
||||
$env: GITHUB_TOKEN
|
||||
- host: ghe.example.net
|
||||
apiBaseUrl: https://ghe.example.net/api/v3
|
||||
rawBaseUrl: https://ghe.example.net/raw
|
||||
token:
|
||||
$env: GHE_TOKEN
|
||||
gitlab:
|
||||
- host: gitlab.com
|
||||
token:
|
||||
$env: GITLAB_TOKEN
|
||||
bitbucket:
|
||||
- host: bitbucket.org
|
||||
username:
|
||||
$env: BITBUCKET_USERNAME
|
||||
appPassword:
|
||||
$env: BITBUCKET_APP_PASSWORD
|
||||
azure:
|
||||
- host: dev.azure.com
|
||||
token:
|
||||
$env: AZURE_TOKEN
|
||||
```
|
||||
|
||||
Each key under `integrations` is a separate configuration for each external
|
||||
provider. The providers each have their own configuration, so let's look at the
|
||||
GitHub section as an example.
|
||||
|
||||
Directly under the `github` key is a list of provider configurations, where you
|
||||
can list the various GitHub compatible providers you want to be able to fetch
|
||||
data from. Each entry is a structure with up to four elements:
|
||||
|
||||
- `host` (optional): The host of the location target that you want to match on.
|
||||
The default host is `github.com`.
|
||||
- `token` (optional): An authentication token as expected by GitHub. If
|
||||
supplied, it will be passed along with all calls to this provider, both API
|
||||
and raw. If it is not supplied, anonymous access will be used.
|
||||
- `apiBaseUrl` (optional): If you want to communicate using the APIv3 method
|
||||
with this provider, specify the base URL for its endpoint here, with no
|
||||
trailing slash. Specifically when the target is GitHub, you can leave it out
|
||||
to be inferred automatically. For a GitHub Enterprise installation, it is
|
||||
commonly at `https://api.<host>` or `https://<host>/api/v3`.
|
||||
- `rawBaseUrl` (optional): If you want to communicate using the raw HTTP method
|
||||
with this provider, specify the base URL for its endpoint here, with no
|
||||
trailing slash. Specifically when the target is public GitHub, you can leave
|
||||
it out to be inferred automatically. For a GitHub Enterprise installation, it
|
||||
is commonly at `https://api.<host>` or `https://<host>/api/v3`.
|
||||
|
||||
You need to supply either `apiBaseUrl` or `rawBaseUrl` or both (except for
|
||||
public GitHub, for which we can infer them). The `apiBaseUrl` will always be
|
||||
preferred over the other if a `token` is given, otherwise `rawBaseUrl` will be
|
||||
preferred.
|
||||
|
||||
If you do not supply a public GitHub provider, one will be added automatically,
|
||||
silently at startup for convenience. So you only have to list it if you want to
|
||||
supply a token for it - and if you do, you can also leave out the `apiBaseUrl`
|
||||
and `rawBaseUrl` fields.
|
||||
|
||||
### Processor: github-discovery
|
||||
|
||||
There is a special GitHub discovery processor for discovering catalog entities
|
||||
within a GitHub organization. The processor will crawl a given GitHub
|
||||
organization and register entities matching the configured path. This can be
|
||||
useful as an alternative to static locations or manually adding things to the
|
||||
catalog.
|
||||
|
||||
To use the discovery processor, you'll need a GitHub integration set up as above
|
||||
with a `GITHUB_TOKEN`. Then you can add a location target to the catalog
|
||||
configuration:
|
||||
|
||||
```yaml
|
||||
catalog:
|
||||
locations:
|
||||
- type: github-discovery
|
||||
target: https://github.com/myorg/service-*/blob/main/catalog-info.yaml
|
||||
```
|
||||
|
||||
Note the `github-discovery` type, as this is not a regular `url` processor.
|
||||
|
||||
The target is composed of three parts:
|
||||
|
||||
- The base organization URL, `https://github.com/myorg` in this case
|
||||
- The repository blob to scan, which accepts \* wildcard tokens. This can simply
|
||||
be `*` to scan all repositories in the organization. This example only looks
|
||||
for repositories prefixed with `service-`.
|
||||
- The path within each repository to find the catalog YAML file. This will
|
||||
usually be `/blob/main/catalog-info.yaml`, `/blob/master/catalog-info.yaml` or
|
||||
a similar variation for catalog files stored in the root directory of each
|
||||
repository.
|
||||
|
||||
## Static Location Configuration
|
||||
|
||||
To enable declarative catalog setups, it is possible to add locations to the
|
||||
catalog via [static configuration](../../conf/index.md). Locations are added to
|
||||
the catalog under the `catalog.locations` key, for example:
|
||||
Locations are added to the catalog under the `catalog.locations` key:
|
||||
|
||||
```yaml
|
||||
catalog:
|
||||
@@ -124,10 +26,35 @@ catalog:
|
||||
target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/artist-lookup-component.yaml
|
||||
```
|
||||
|
||||
The locations added through static configuration can not be removed through the
|
||||
catalog locations API. To remove the locations, you have to remove them from the
|
||||
The `url` type locations are handled by a standard processor included with the
|
||||
catalog (`UrlReaderProcessor`), so no processor configuration is needed. This
|
||||
processor _does however_ need an [integration](../../integrations/index.md) to
|
||||
understand how to retrieve a given URL. For the example above, you would need to
|
||||
configure the [GitHub integration](../../integrations/github/locations.md) to
|
||||
read files from github.com.
|
||||
|
||||
The locations added through static configuration cannot be removed through the
|
||||
catalog locations API. To remove these locations, you must remove them from the
|
||||
configuration.
|
||||
|
||||
### Integration Processors
|
||||
|
||||
Integrations may simply provide a mechanism to handle `url` location type for an
|
||||
external provider, or they may also include additional processors out of the
|
||||
box, such as the GitHub [discovery](../../integrations/github/discovery.md)
|
||||
processor that scans a GitHub organization for
|
||||
[entity descriptor files](descriptor-format.md).
|
||||
|
||||
Check the [integrations](../../integrations/index.md) documentation to see what
|
||||
is offered by each integration.
|
||||
|
||||
### Custom Processors
|
||||
|
||||
To ingest entities from an existing system already tracking software, you can
|
||||
also write a _custom processor_ to convert between the existing system and
|
||||
Backstage's descriptor format. This is documented in
|
||||
[External Integrations](external-integrations.md).
|
||||
|
||||
## Catalog Rules
|
||||
|
||||
By default the catalog will only allow ingestion of entities with the kind
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
---
|
||||
id: locations
|
||||
title: Azure DevOps Locations
|
||||
sidebar_label: Locations
|
||||
description: Documentation on Azure DevOps location integration
|
||||
---
|
||||
|
||||
The Azure integration supports loading catalog entities from Azure DevOps.
|
||||
Components can be added to
|
||||
[static catalog configuration](../../features/software-catalog/configuration.md),
|
||||
or registered with the
|
||||
[catalog-import](https://github.com/backstage/backstage/tree/master/plugins/catalog-import)
|
||||
plugin.
|
||||
|
||||
```yaml
|
||||
integrations:
|
||||
azure:
|
||||
- host: dev.azure.com
|
||||
token:
|
||||
$env: AZURE_TOKEN
|
||||
```
|
||||
|
||||
> Note: An Azure DevOps provider is added automatically at startup for
|
||||
> convenience, so you only need to list it if you want to supply a
|
||||
> [token](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate).
|
||||
|
||||
The configuration is a structure with two elements:
|
||||
|
||||
- `host`: The DevOps host; only `dev.azure.com` is supported.
|
||||
- `token` (optional): An personal access token as expected by Azure DevOps.
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
id: locations
|
||||
title: BitBucket Locations
|
||||
sidebar_label: Locations
|
||||
description: Documentation on BitBucket location integration
|
||||
---
|
||||
|
||||
The BitBucket integration supports loading catalog entities from bitbucket.com
|
||||
or a self-hosted BitBucket. Components can be added to
|
||||
[static catalog configuration](../../features/software-catalog/configuration.md),
|
||||
or registered with the
|
||||
[catalog-import](https://github.com/backstage/backstage/tree/master/plugins/catalog-import)
|
||||
plugin.
|
||||
|
||||
```yaml
|
||||
integrations:
|
||||
bitbucket:
|
||||
- host: bitbucket.org
|
||||
username:
|
||||
$env: BITBUCKET_USERNAME
|
||||
token:
|
||||
$env: BITBUCKET_TOKEN
|
||||
```
|
||||
|
||||
> Note: A public BitBucket provider is added automatically at startup for
|
||||
> convenience, so you only need to list it if you want to supply a
|
||||
> [token](https://confluence.atlassian.com/bitbucketserver/personal-access-tokens-939515499.html).
|
||||
|
||||
Directly under the `bitbucket` key is a list of provider configurations, where
|
||||
you can list the BitBucket providers you want to fetch data from. Each entry is
|
||||
a structure with up to four elements:
|
||||
|
||||
- `host`: The host of the BitBucket instance, e.g. `bitbucket.company.com`.
|
||||
- `username`: The BitBucket username to use in API requests. If a username is
|
||||
not supplied, anonymous access will be used.
|
||||
- `token` (optional): An personal access token as expected by BitBucket. Either
|
||||
an access token **or** a password may be supplied.
|
||||
- `appPassword` (optional): The password for the BitBucket user. Either an
|
||||
appPassword **or** an access token may be supplied.
|
||||
- `apiBaseUrl` (optional): The URL of the GitLab API. For self-hosted
|
||||
installations, it is commonly at `https://<host>/api/v4`. For gitlab.com, this
|
||||
configuration is not needed as it can be inferred.
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
id: discovery
|
||||
title: GitHub Discovery
|
||||
sidebar_label: Discovery
|
||||
description: Documentation on GitHub organization discovery
|
||||
---
|
||||
|
||||
The GitHub integration has a special discovery processor for discovering catalog
|
||||
entities within a GitHub organization. The processor will crawl the GitHub
|
||||
organization and register entities matching the configured path. This can be
|
||||
useful as an alternative to static locations or manually adding things to the
|
||||
catalog.
|
||||
|
||||
To use the discovery processor, you'll need a GitHub integration
|
||||
[set up](locations.md) with a `GITHUB_TOKEN`. Then you can add a location target
|
||||
to the catalog configuration:
|
||||
|
||||
```yaml
|
||||
catalog:
|
||||
locations:
|
||||
- type: github-discovery
|
||||
target: https://github.com/myorg/service-*/blob/main/catalog-info.yaml
|
||||
```
|
||||
|
||||
Note the `github-discovery` type, as this is not a regular `url` processor.
|
||||
|
||||
The target is composed of three parts:
|
||||
|
||||
- The base organization URL, `https://github.com/myorg` in this case
|
||||
- The repository blob to scan, which accepts \* wildcard tokens. This can simply
|
||||
be `*` to scan all repositories in the organization. This example only looks
|
||||
for repositories prefixed with `service-`.
|
||||
- The path within each repository to find the catalog YAML file. This will
|
||||
usually be `/blob/main/catalog-info.yaml`, `/blob/master/catalog-info.yaml` or
|
||||
a similar variation for catalog files stored in the root directory of each
|
||||
repository.
|
||||
|
||||
## GitHub API Rate Limits
|
||||
|
||||
GitHub
|
||||
[rate limits](https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting)
|
||||
API requests to 5,000 per hour (or more for Enterprise accounts). The default
|
||||
Backstage catalog backend refreshes data every 100 seconds, which issues an API
|
||||
request for each discovered location.
|
||||
|
||||
This means if you have more than ~140 catalog entities, you may get throttled by
|
||||
rate limiting. This will soon be resolved once catalog refreshes make use of
|
||||
ETags; to work around this in the meantime, you can change the refresh rate of
|
||||
the catalog in your `packages/backend/src/plugins/catalog.ts` file.
|
||||
|
||||
This is true for any method of adding GitHub entities to the catalog, but
|
||||
especially easy to hit with automatic discovery.
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
id: locations
|
||||
title: GitHub Locations
|
||||
sidebar_label: Locations
|
||||
description: Documentation on GitHub location integration
|
||||
---
|
||||
|
||||
The GitHub integration supports loading catalog entities from github.com or
|
||||
GitHub Enterprise. Components can be added to
|
||||
[static catalog configuration](../../features/software-catalog/configuration.md),
|
||||
registered with the
|
||||
[catalog-import](https://github.com/backstage/backstage/tree/master/plugins/catalog-import)
|
||||
plugin, or [discovered](discovery.md) from a GitHub organization. Users and
|
||||
Groups can also be [loaded from an organization](org.md).
|
||||
|
||||
## Configuration
|
||||
|
||||
To use this integration, add configuration to your root `app-config.yaml`:
|
||||
|
||||
```yaml
|
||||
integrations:
|
||||
github:
|
||||
- host: github.com
|
||||
token:
|
||||
$env: GITHUB_TOKEN
|
||||
- host: ghe.example.net
|
||||
apiBaseUrl: https://ghe.example.net/api/v3
|
||||
rawBaseUrl: https://ghe.example.net/raw
|
||||
token:
|
||||
$env: GHE_TOKEN
|
||||
```
|
||||
|
||||
> Note: A public GitHub provider is added automatically at startup for
|
||||
> convenience, so you only need to list it if you want to supply a
|
||||
> [token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token).
|
||||
|
||||
Directly under the `github` key is a list of provider configurations, where you
|
||||
can list the various GitHub-compatible providers you want to be able to fetch
|
||||
data from. Each entry is a structure with up to four elements:
|
||||
|
||||
- `host` (optional): The host of the location target that you want to match on.
|
||||
The default host is `github.com`.
|
||||
- `token` (optional): An authentication token as expected by GitHub. If
|
||||
supplied, it will be passed along with all calls to this provider, both API
|
||||
and raw. If it is not supplied, anonymous access will be used.
|
||||
- `apiBaseUrl` (optional): If you want to communicate using the APIv3 method
|
||||
with this provider, specify the base URL for its endpoint here, with no
|
||||
trailing slash. Specifically when the target is GitHub, you can leave it out
|
||||
to be inferred automatically. For a GitHub Enterprise installation, it is
|
||||
commonly at `https://api.<host>` or `https://<host>/api/v3`.
|
||||
- `rawBaseUrl` (optional): If you want to communicate using the raw HTTP method
|
||||
with this provider, specify the base URL for its endpoint here, with no
|
||||
trailing slash. Specifically when the target is public GitHub, you can leave
|
||||
it out to be inferred automatically. For a GitHub Enterprise installation, it
|
||||
is commonly at `https://api.<host>` or `https://<host>/api/v3`.
|
||||
|
||||
You need to supply either `apiBaseUrl` or `rawBaseUrl` or both (except for
|
||||
public GitHub, for which we can infer them). The `apiBaseUrl` will always be
|
||||
preferred over the other if a `token` is given, otherwise `rawBaseUrl` will be
|
||||
preferred.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
id: locations
|
||||
title: GitLab Locations
|
||||
sidebar_label: Locations
|
||||
description: Documentation on GitLab location integration
|
||||
---
|
||||
|
||||
The GitLab integration supports loading catalog entities from gitlab.com or a
|
||||
self-hosted GitLab. Components can be added to
|
||||
[static catalog configuration](../../features/software-catalog/configuration.md),
|
||||
or registered with the
|
||||
[catalog-import](https://github.com/backstage/backstage/tree/master/plugins/catalog-import)
|
||||
plugin.
|
||||
|
||||
```yaml
|
||||
integrations:
|
||||
gitlab:
|
||||
- host: gitlab.com
|
||||
token:
|
||||
$env: GITLAB_TOKEN
|
||||
```
|
||||
|
||||
> Note: A public GitLab provider is added automatically at startup for
|
||||
> convenience, so you only need to list it if you want to supply a
|
||||
> [token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html).
|
||||
|
||||
Directly under the `gitlab` key is a list of provider configurations, where you
|
||||
can list the GitLab providers you want to fetch data from. Each entry is a
|
||||
structure with up to four elements:
|
||||
|
||||
- `host`: The host of the GitLab instance, e.g. `gitlab.company.com`.
|
||||
- `token` (optional): An authentication token as expected by GitLab. If
|
||||
supplied, it will be passed along with all calls to this provider, both API
|
||||
and raw. If it is not supplied, anonymous access will be used.
|
||||
- `apiBaseUrl` (optional): The URL of the GitLab API. For self-hosted
|
||||
installations, it is commonly at `https://<host>/api/v4`. For gitlab.com, this
|
||||
configuration is not needed as it can be inferred.
|
||||
- `baseUrl` (optional): The base URL for this provider, e.g.
|
||||
`https://gitlab.com`. If this is not provided, it is assumed to be
|
||||
`https://{host}`.
|
||||
@@ -0,0 +1,36 @@
|
||||
---
|
||||
id: index
|
||||
title: Integrations
|
||||
sidebar_label: Overview
|
||||
description: Documentation on Backstage Integrations
|
||||
---
|
||||
|
||||
Integrations allow Backstage to read or publish data using external providers
|
||||
|
||||
- such as GitHub, GitLab, BitBucket, LDAP, or cloud providers.
|
||||
|
||||
## Configuration
|
||||
|
||||
Integration are configured at the root level of `app-config.yaml` since
|
||||
integrations are used by many Backstage core features and other plugins.
|
||||
|
||||
Each key under `integrations` is a separate configuration for a single external
|
||||
provider. Providers each have different configuration; here's an example of
|
||||
configuration to use both GitHub and BitBucket:
|
||||
|
||||
```yaml
|
||||
integrations:
|
||||
github:
|
||||
- host: github.com
|
||||
token:
|
||||
$env: GITHUB_TOKEN
|
||||
bitbucket:
|
||||
- host: bitbucket.org
|
||||
username:
|
||||
$env: BITBUCKET_USERNAME
|
||||
appPassword:
|
||||
$env: BITBUCKET_APP_PASSWORD
|
||||
```
|
||||
|
||||
See documentation for each type of integration for full details on
|
||||
configuration.
|
||||
+27
-1
@@ -104,10 +104,36 @@
|
||||
}
|
||||
],
|
||||
"Integrations": [
|
||||
"integrations/index",
|
||||
{
|
||||
"type": "subcategory",
|
||||
"label": "GitHub",
|
||||
"ids": ["integrations/github/org"]
|
||||
"ids": [
|
||||
"integrations/github/locations",
|
||||
"integrations/github/discovery",
|
||||
"integrations/github/org"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "subcategory",
|
||||
"label": "GitLab",
|
||||
"ids": [
|
||||
"integrations/gitlab/locations"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "subcategory",
|
||||
"label": "BitBucket",
|
||||
"ids": [
|
||||
"integrations/bitbucket/locations"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "subcategory",
|
||||
"label": "Azure DevOps",
|
||||
"ids": [
|
||||
"integrations/azure/locations"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "subcategory",
|
||||
|
||||
Reference in New Issue
Block a user