paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 22 Packages Projects Releases Wiki Activity

More Backend Migration Docs

Browse Source 
* Added more documentation on migrating catalog modules to the new backend system.  (A-Github)
* Tweaked a few existing documentation page to recommend schedules are set via

Signed-off-by: Alex Crome <afscrome@users.noreply.github.com>
This commit is contained in:
Alex Crome
2023-11-15 22:34:11 +00:00
parent 68f1ff104c
commit 6fb909f011
7 changed files with 269 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/backend-system/building-backends/08-migrating.md
+254 -4
View File
@@ -262,9 +262,259 @@ If you have other customizations made to `plugins/catalog.ts`, such as adding
custom processors or entity providers, read on. Otherwise, you should be able to
just delete that file at this point.
#### Amazon Web Services
`AwsEksClusterProcessor` and `AwsOrganizationCloudAccountProcessor` have not yet been migrated to the new backend system.
See [Other Catalog Extensions](#other-catalog-extensions) for how to use these in the new backend system.
For `AwsS3DiscoveryProcessor`, first migrate to `AwsS3EntityProvider`.
To migrate `AwsS3EntityProvider` to the new backend system, add a reference to the `@backstage/plugin-catalog-backend-module-aws` module.
```ts title="packages/backend/src/index.ts"
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
/* highlight-add-start */
backend.add(import('@backstage/plugin-catalog-backend-module-aws/alpha'));
/* highlight-add-end */
```
If you were providing a `schedule` in code, this now needs to be set via configuration.
All other AWS configuration in `app-config.yaml` remains the same.
```yaml title="app-config.yaml"
catalog:
providers:
awsS3:
yourProviderId:
# ...
/* highlight-add-start */
schedule:
frequency: PT1H
timeout: PT50M
/* highlight-add-end */
```
#### Azure Devops
For `AzureDevOpsDiscoveryProcessor`, first migrate to `AzureDevOpsEntityProvider`.
To migrate `AzureDevOpsEntityProvider` to the new backend system, add a reference to the `@backstage/plugin-catalog-backend-module-azure` module.
```ts title="packages/backend/src/index.ts"
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
/* highlight-add-start */
backend.add(import('@backstage/plugin-catalog-backend-module-azure/alpha'));
/* highlight-add-end */
```
If you were providing a `schedule` in code, this now needs to be set via configuration.
All other Azure DevOps configuration in `app-config.yaml` remains the same.
```yaml title="app-config.yaml"
catalog:
providers:
azureDevOps:
yourProviderId:
# ...
/* highlight-add-start */
schedule:
frequency: PT1H
timeout: PT50M
/* highlight-add-end */
```
#### Open API
`InternalOpenApiDocumentationProvider` have not yet been migrated to the new backend system.
See [Other Catalog Extensions](#other-catalog-extensions) for how to use this in the new backend system.
#### Bitbucket
For `BitbucketDiscoveryProcessor`, migrate to `BitbucketCloudEntityProvider` or `BitbucketServerEntityProvider`
To migrate `BitbucketCloudEntityProvider` to the new backend system, add a reference to the `@backstage/plugin-catalog-backend-module-bitbucket-cloud` module.
```ts title="packages/backend/src/index.ts"
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
/* highlight-add-start */
backend.add(
import('@backstage/plugin-catalog-backend-module-bitbucket-cloud/alpha'),
);
/* highlight-add-end */
```
If you were providing a `schedule` in code, this now needs to be set via configuration.
All other Bitbucket Cloud configuration in `app-config.yaml` remains the same.
```yaml title="app-config.yaml"
catalog:
providers:
bitbucketCloud:
yourProviderId:
# ...
/* highlight-add-start */
schedule:
frequency: PT30M
timeout: PT3M
/* highlight-add-end */
```
To migrate `BitbucketServerEntityProvider` to the new backend system, add a reference to `@backstage/plugin-catalog-backend-module-bitbucket-server`.
```ts title="packages/backend/src/index.ts"
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
/* highlight-add-start */
backend.add(
import('@backstage/plugin-catalog-backend-module-bitbucket-server/alpha'),
);
/* highlight-add-end */
```
If you were providing a `schedule` in code, this now needs to be set via configuration.
All other Bitbucket Server configuration in `app-config.yaml` remains the same.
```yaml title="app-config.yaml"
catalog:
providers:
bitbucketServer:
yourProviderId:
# ...
/* highlight-add-start */
schedule:
frequency: PT30M
timeout: PT3M
/* highlight-add-end */
```
#### Google Cloud Platform
To migrate `GkeEntityProvider` to the new backend system, add a reference to `@backstage/plugin-catalog-backend-module-gcp`.
```ts title="packages/backend/src/index.ts"
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
/* highlight-add-start */
backend.add(import('@backstage/plugin-catalog-backend-module-gcp'));
/* highlight-add-end */
```
Configuration in app-config.yaml remains the same.
#### Gerrit
To migrate `GerritEntityProvider` to the new backend system, add a reference to `@backstage/plugin-catalog-backend-module-gerrit`.
```ts title="packages/backend/src/index.ts"
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
/* highlight-add-start */
backend.add(import('@backstage/plugin-catalog-backend-module-gerrit/alpha'));
/* highlight-add-end */
```
If you were providing a `schedule` in code, this now needs to be set via configuration.
All other Gerrit configuration in `app-config.yaml` remains the same.
```yaml title="app-config.yaml"
catalog:
providers:
gerrit:
yourProviderId:
# ...
/* highlight-add-start */
schedule:
frequency: PT30M
timeout: PT3M
/* highlight-add-end */
```
#### Github
For `GithubDiscoveryProcessor`, `GithubMultiOrgReaderProcessor` and `GithubOrgReaderProcessor`, first migrate to the equivalent Entity Provider.
To migrate `GithubEntityProvider` to the new backend system, add a reference to `@backstage/plugin-catalog-backend-module-github`.
```ts title="packages/backend/src/index.ts"
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
/* highlight-add-start */
backend.add(import('@backstage/plugin-catalog-backend-module-github/alpha'));
/* highlight-add-end */
```
If you were providing a `schedule` in code, this now needs to be set via configuration.
All other Github configuration in `app-config.yaml` remains the same.
```yaml title="app-config.yaml"
catalog:
providers:
github:
yourProviderId:
# ...
/* highlight-add-start */
schedule:
frequency: PT30M
timeout: PT3M
/* highlight-add-end */
```
To migrate `GithubMultiOrgEntityProvider` and `GithubOrgEntityProvider` to the new backend system, add a reference to `@backstage/plugin-catalog-backend-module-github-org`.
```ts title="packages/backend/src/index.ts"
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
/* highlight-add-start */
backend.add(import('@backstage/plugin-catalog-backend-module-github-org'));
/* highlight-add-end */
```
If you were providing a `schedule` in code, this now needs to be set via configuration.
All other Github configuration in `app-config.yaml` remains the same.
```yaml title="app-config.yaml"
catalog:
providers:
githubOrg:
yourProviderId:
# ...
/* highlight-add-start */
schedule:
frequency: PT30M
timeout: PT3M
/* highlight-add-end */
```
If you were providing transformers, these can be configured by extending `githubOrgEntityProviderTransformsExtensionPoint`
```ts title="packages/backend/src/index.ts"
import { createBackendModule } from '@backstage/backend-plugin-api';
import { githubOrgEntityProviderTransformsExtensionPoint } from '@backstage/plugin-catalog-backend-module-github-org';
backend.add(
createBackendModule({
pluginId: 'catalog',
moduleId: 'githubOrgTransformers',
register(env) {
env.registerInit({
deps: {
/* highlight-add-start */
githubOrgTransformers:
githubOrgEntityProviderTransformsExtensionPoint,
/* highlight-add-end */
},
async init({ githubOrgTransformers }) {
/* highlight-add-start */
githubOrgTransformers.setUserTransformer(myUserTransformer);
githubOrgTransformers.setTeamTransformer(myTeamTransformer);
/* highlight-add-end */
},
});
},
}),
);
```
#### Microsoft Graph
Import the Microsoft Graph catalog module
For `MicrosoftGraphOrgReaderProcessor`, first migrate to `MicrosoftGraphOrgEntityProvider`
To migrate `MicrosoftGraphOrgEntityProvider` to the new backend system, add a reference to `@backstage/plugin-catalog-backend-module-github-org`.
```ts title="packages/backend/src/index.ts"
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
@@ -273,7 +523,8 @@ backend.add(import('@backstage/plugin-catalog-backend-module-msgraph/alpha'));
/* highlight-add-end */
```
If you were providng a `schedule` programtically, this now needs to be set via configuration
If you were providing a `schedule` in code, this now needs to be set via configuration
All other Microsoft Graph configuration in `app-config.yaml` remains the same.
```yaml title="app-config.yaml"
catalog:
@@ -285,7 +536,6 @@ catalog:
frequency: PT4H
timeout: PT30M
/* highlight-add-end */
```
If you were providing transformers, these can be configured by extending `microsoftGraphOrgEntityProviderTransformExtensionPoint`
@@ -297,7 +547,7 @@ import { microsoftGraphOrgEntityProviderTransformExtensionPoint } from '@backsta
backend.add(
createBackendModule({
pluginId: 'catalog',
moduleId: 'microsoftGraphExtensions',
moduleId: 'microsoftGraphTransformers',
register(env) {
env.registerInit({
deps: {
docs/integrations/aws-s3/discovery.md
+2 -8
View File
@@ -32,7 +32,7 @@ catalog:
bucketName: sample-bucket
prefix: prefix/ # optional
region: us-east-2 # optional, uses the default region otherwise
schedule: # optional; same options as in TaskScheduleDefinition
schedule: # same options as in TaskScheduleDefinition
# supports cron, ISO duration, "human duration" as used in code
frequency: { minutes: 30 }
# supports ISO duration, "human duration" as used in code
@@ -52,7 +52,7 @@ catalog:
bucketName: sample-bucket
prefix: prefix/ # optional
region: us-east-2 # optional, uses the default region otherwise
schedule: # optional; same options as in TaskScheduleDefinition
schedule: # same options as in TaskScheduleDefinition
# supports cron, ISO duration, "human duration" as used in code
frequency: { minutes: 30 }
# supports ISO duration, "human duration" as used in code
@@ -79,12 +79,6 @@ const builder = await CatalogBuilder.create(env);
builder.addEntityProvider(
AwsS3EntityProvider.fromConfig(env.config, {
logger: env.logger,
// optional: alternatively, use scheduler with schedule defined in app-config.yaml
schedule: env.scheduler.createScheduledTaskRunner({
frequency: { minutes: 30 },
timeout: { minutes: 3 },
}),
// optional: alternatively, use schedule
scheduler: env.scheduler,
}),
);
docs/integrations/azure/discovery.md
+1 -1
View File
@@ -71,7 +71,7 @@ The parameters available are:
- **`repository:`** _(optional)_ The repository name. Wildcards are supported as show on the examples above. If not set, all repositories will be searched.
- **`path:`** _(optional)_ Where to find catalog-info.yaml files. Defaults to /catalog-info.yaml.
- **`branch:`** _(optional)_ The branch name to use.
- **`schedule`** _(optional)_:
- **`schedule`**:
- **`frequency`**:
How often you want the task to run. The system does its best to avoid overlapping invocations.
- **`timeout`**:
docs/integrations/bitbucketCloud/discovery.md
+2 -15
View File
@@ -49,19 +49,6 @@ export default async function createPlugin(
}
```
Alternatively to the config-based schedule, you can use
```ts
/* highlight-remove-next-line */
scheduler: env.scheduler,
/* highlight-add-start */
schedule: env.scheduler.createScheduledTaskRunner({
frequency: { minutes: 30 },
timeout: { minutes: 3 },
}),
/* highlight-add-end */
```
### Installation with Events Support
Please follow the installation instructions at
@@ -131,7 +118,7 @@ catalog:
filters: # optional
projectKey: '^apis-.*$' # optional; RegExp
repoSlug: '^service-.*$' # optional; RegExp
schedule: # optional; same options as in TaskScheduleDefinition
schedule: # same options as in TaskScheduleDefinition
# supports cron, ISO duration, "human duration" as used in code
frequency: { minutes: 30 }
# supports ISO duration, "human duration" as used in code
@@ -153,7 +140,7 @@ catalog:
Regular expression used to filter results based on the project key.
- **`repoSlug`** _(optional)_:
Regular expression used to filter results based on the repo slug.
- **`schedule`** _(optional)_:
- **`schedule`**:
- **`frequency`**:
How often you want the task to run. The system does its best to avoid overlapping invocations.
- **`timeout`**:
docs/integrations/bitbucketServer/discovery.md
+2 -8
View File
@@ -38,12 +38,6 @@ export default async function createPlugin(
builder.addEntityProvider(
BitbucketServerEntityProvider.fromConfig(env.config, {
logger: env.logger,
// optional: alternatively, use scheduler with schedule defined in app-config.yaml
schedule: env.scheduler.createScheduledTaskRunner({
frequency: { minutes: 30 },
timeout: { minutes: 3 },
}),
// optional: alternatively, use schedule
scheduler: env.scheduler,
}),
);
@@ -69,7 +63,7 @@ catalog:
filters: # optional
projectKey: '^apis-.*$' # optional; RegExp
repoSlug: '^service-.*$' # optional; RegExp
schedule: # optional; same options as in TaskScheduleDefinition
schedule: # same options as in TaskScheduleDefinition
# supports cron, ISO duration, "human duration" as used in code
frequency: { minutes: 30 }
# supports ISO duration, "human duration" as used in code
@@ -87,7 +81,7 @@ catalog:
Regular expression used to filter results based on the project key.
- **`repoSlug`** _(optional)_:
Regular expression used to filter results based on the repo slug.
- **`schedule`** _(optional)_:
- **`schedule`**:
- **`frequency`**:
How often you want the task to run. The system does its best to avoid overlapping invocations.
- **`timeout`**:
docs/integrations/gerrit/discovery.md
+1 -8
View File
@@ -26,18 +26,11 @@ Then add the plugin to the plugin catalog `packages/backend/src/plugins/catalog.
```ts
/* packages/backend/src/plugins/catalog.ts */
import { GerritEntityProvider } from '@backstage/plugin-catalog-backend-module-gerrit';
import { Duration } from 'luxon';
const builder = await CatalogBuilder.create(env);
/** ... other processors and/or providers ... */
builder.addEntityProvider(
GerritEntityProvider.fromConfig(env.config, {
logger: env.logger,
// optional: alternatively, use scheduler with schedule defined in app-config.yaml
schedule: env.scheduler.createScheduledTaskRunner({
frequency: { minutes: 30 },
timeout: { minutes: 3 },
}),
// optional: alternatively, use schedule
scheduler: env.scheduler,
}),
);
@@ -57,7 +50,7 @@ catalog:
host: gerrit-your-company.com
branch: master # Optional
query: 'state=ACTIVE&prefix=webapps'
schedule: # optional; same options as in TaskScheduleDefinition
schedule:
# supports cron, ISO duration, "human duration" as used in code
frequency: { minutes: 30 }
# supports ISO duration, "human duration" as used in code
docs/integrations/github/discovery.md
+7 -20
View File
@@ -40,12 +40,6 @@ export default async function createPlugin(
builder.addEntityProvider(
GithubEntityProvider.fromConfig(env.config, {
logger: env.logger,
// optional: alternatively, use scheduler with schedule defined in app-config.yaml
schedule: env.scheduler.createScheduledTaskRunner({
frequency: { minutes: 30 },
timeout: { minutes: 3 },
}),
// optional: alternatively, use schedule
scheduler: env.scheduler,
}),
);
@@ -85,12 +79,6 @@ export default async function createPlugin(
/* highlight-add-start */
const githubProvider = GithubEntityProvider.fromConfig(env.config, {
logger: env.logger,
// optional: alternatively, use scheduler with schedule defined in app-config.yaml
schedule: env.scheduler.createScheduledTaskRunner({
frequency: { minutes: 30 },
timeout: { minutes: 3 },
}),
// optional: alternatively, use schedule
scheduler: env.scheduler,
});
env.eventBroker.subscribe(githubProvider);
@@ -122,7 +110,7 @@ catalog:
filters:
branch: 'main' # string
repository: '.*' # Regex
schedule: # optional; same options as in TaskScheduleDefinition
schedule: # same options as in TaskScheduleDefinition
# supports cron, ISO duration, "human duration" as used in code
frequency: { minutes: 30 }
# supports ISO duration, "human duration" as used in code
@@ -213,7 +201,7 @@ This provider supports multiple organizations via unique provider IDs.
Defaults to `false`.
Due to limitations in the GitHub API's ability to query for repository objects, this option cannot be used in
conjunction with wildcards in the `catalogPath`.
- **`schedule`** _(optional)_:
- **`schedule`**:
- **`frequency`**:
How often you want the task to run. The system does its best to avoid overlapping invocations.
- **`timeout`**:
@@ -229,13 +217,12 @@ GitHub [rate limits](https://docs.github.com/en/rest/overview/resources-in-the-r
accounts). The snippet below refreshes the Backstage catalog data every 35 minutes, which issues an API request for each discovered location.
If your requests are too frequent then you may get throttled by
rate limiting. You can change the refresh frequency of the catalog in your `packages/backend/src/plugins/catalog.ts` file:
rate limiting. You can change the refresh frequency of the catalog in your `app-config.yaml` file by controlling the `schedule`.
```typescript
schedule: env.scheduler.createScheduledTaskRunner({
frequency: { minutes: 35 },
timeout: { minutes: 30 },
}),
```yaml
schedule:
frequency: { minutes: 35 }
timeout: { minutes: 3 }
```
More information about scheduling can be found on the [TaskScheduleDefinition](https://backstage.io/docs/reference/backend-tasks.taskscheduledefinition) page.