paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 35 Packages Projects Releases Wiki Activity

Merge pull request #14034 from Bonial-International-GmbH/pjungermann/schedule-in-config

Browse Source 
feat(catalog/[...]): schedule via config + backend-plugin-api support
This commit is contained in:
Fredrik Adelöw
2022-10-19 12:55:40 +01:00
committed by GitHub
parent f077228e0c 6bb046bcbe
commit 545d64cbc9
69 changed files with 1913 additions and 154 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.changeset/chatty-planets-flash.md
+5
View File
@@ -0,0 +1,5 @@
---
'@backstage/plugin-catalog-backend-module-gitlab': patch
---
Add `gitlabDiscoveryEntityProviderCatalogModule` (new backend-plugin-api, alpha).
.changeset/clean-planets-rhyme.md
+8
View File
@@ -0,0 +1,8 @@
---
'@backstage/plugin-catalog-backend-module-azure': patch
---
`AzureDevOpsEntityProvider`: Add option to configure schedule via `app-config.yaml` instead of in code.
Please find how to configure the schedule at the config at
https://backstage.io/docs/integrations/azure/discovery
.changeset/flat-kangaroos-kiss.md
+8
View File
@@ -0,0 +1,8 @@
---
'@backstage/plugin-catalog-backend-module-gitlab': patch
---
`GitlabDiscoveryEntityProvider`: Add option to configure schedule via `app-config.yaml` instead of in code.
Please find how to configure the schedule at the config at
https://backstage.io/docs/integrations/gitlab/discovery
.changeset/gorgeous-onions-thank.md
+8
View File
@@ -0,0 +1,8 @@
---
'@backstage/plugin-catalog-backend-module-aws': patch
---
`AwsS3EntityProvider`: Add option to configure schedule via `app-config.yaml` instead of in code.
Please find how to configure the schedule at the config at
https://backstage.io/docs/integrations/aws-s3/discovery
.changeset/itchy-paws-protect.md
+5
View File
@@ -0,0 +1,5 @@
---
'@backstage/plugin-catalog-backend-module-azure': patch
---
Add `azureDevOpsEntityProviderCatalogModule` (new backend-plugin-api, alpha).
.changeset/kind-emus-juggle.md
+5
View File
@@ -0,0 +1,5 @@
---
'@backstage/plugin-catalog-backend-module-aws': patch
---
Add `awsS3EntityProviderCatalogModule` (new backend-plugin-api, alpha).
.changeset/metal-dogs-swim.md
+5
View File
@@ -0,0 +1,5 @@
---
'@backstage/plugin-catalog-backend-module-gerrit': patch
---
Add `gerritEntityProviderCatalogModule` (new backend-plugin-api, alpha).
.changeset/tasty-scissors-tickle.md
+8
View File
@@ -0,0 +1,8 @@
---
'@backstage/plugin-catalog-backend-module-bitbucket-server': patch
---
`BitbucketServerEntityProvider`: Add option to configure schedule via `app-config.yaml` instead of in code.
Please find how to configure the schedule at the config at
https://backstage.io/docs/integrations/bitbucketServer/discovery
.changeset/ten-pens-draw.md
+8
View File
@@ -0,0 +1,8 @@
---
'@backstage/plugin-catalog-backend-module-gerrit': patch
---
`GerritEntityProvider`: Add option to configure schedule via `app-config.yaml` instead of in code.
Please find how to configure the schedule at the config at
https://backstage.io/docs/integrations/gerrit/discovery
.changeset/two-oranges-joke.md
+5
View File
@@ -0,0 +1,5 @@
---
'@backstage/plugin-catalog-backend-module-bitbucket-server': patch
---
Add `bitbucketServerEntityProviderCatalogModule` (new backend-plugin-api, alpha).
docs/integrations/aws-s3/discovery.md
+13
View File
@@ -32,6 +32,11 @@ catalog:
bucketName: sample-bucket
prefix: prefix/ # optional
region: us-east-2 # optional, uses the default region otherwise
schedule: # optional; same options as in TaskScheduleDefinition
# supports cron, ISO duration, "human duration" as used in code
frequency: { minutes: 30 }
# supports ISO duration, "human duration" as used in code
timeout: { minutes: 3 }
```
For simple setups, you can omit the provider ID at the config
@@ -47,6 +52,11 @@ catalog:
bucketName: sample-bucket
prefix: prefix/ # optional
region: us-east-2 # optional, uses the default region otherwise
schedule: # optional; same options as in TaskScheduleDefinition
# supports cron, ISO duration, "human duration" as used in code
frequency: { minutes: 30 }
# supports ISO duration, "human duration" as used in code
timeout: { minutes: 3 }
```
As this provider is not one of the default providers, you will first need to install
@@ -69,10 +79,13 @@ const builder = await CatalogBuilder.create(env);
builder.addEntityProvider(
AwsS3EntityProvider.fromConfig(env.config, {
logger: env.logger,
// optional: alternatively, use scheduler with schedule defined in app-config.yaml
schedule: env.scheduler.createScheduledTaskRunner({
frequency: { minutes: 30 },
timeout: { minutes: 3 },
}),
// optional: alternatively, use schedule
scheduler: env.scheduler,
}),
);
```
docs/integrations/azure/discovery.md
+25 -8
View File
@@ -42,12 +42,17 @@ catalog:
project: myproject
repository: service-* # this will match all repos starting with service-*
path: /catalog-info.yaml
schedule: # optional; same options as in TaskScheduleDefinition
# supports cron, ISO duration, "human duration" as used in code
frequency: { minutes: 30 }
# supports ISO duration, "human duration" as used in code
timeout: { minutes: 3 }
anotherProviderId: # another identifier
organization: myorg
project: myproject
repository: '*' # this will match all repos
path: /src/*/catalog-info.yaml # this will search for files deep inside the /src folder
yetAotherProviderId: # guess, what? Another one :)
yetAnotherProviderId: # guess, what? Another one :)
host: selfhostedazure.yourcompany.com
organization: myorg
project: myproject
@@ -55,11 +60,20 @@ catalog:
The parameters available are:
- `host:` Leave empty for Cloud hosted, otherwise set to your self-hosted instance host.
- `organization:` Your Organization slug (or Collection for on-premise users). Required.
- `project:` Your project slug. Required.
- `repository:` The repository name. Wildcards are supported as show on the examples above. If not set, all repositories will be searched.
- `path:` Where to find catalog-info.yaml files. Defaults to /catalog-info.yaml.
- **`host:`** _(optional)_ Leave empty for Cloud hosted, otherwise set to your self-hosted instance host.
- **`organization:`** Your Organization slug (or Collection for on-premise users). Required.
- **`project:`** Your project slug. Required.
- **`repository:`** _(optional)_ The repository name. Wildcards are supported as show on the examples above. If not set, all repositories will be searched.
- **`path:`** _(optional)_ Where to find catalog-info.yaml files. Defaults to /catalog-info.yaml.
- **`schedule`** _(optional)_:
- **`frequency`**:
How often you want the task to run. The system does its best to avoid overlapping invocations.
- **`timeout`**:
The maximum amount of time that a single task invocation can take.
- **`initialDelay`** _(optional)_:
The amount of time that should pass before the first invocation happens.
- **`scope`** _(optional)_:
`'global'` or `'local'`. Sets the scope of concurrency control.
_Note:_ the path parameter follows the same rules as the search on Azure DevOps
web interface. For more details visit the
@@ -84,10 +98,13 @@ const builder = await CatalogBuilder.create(env);
+builder.addEntityProvider(
+ AzureDevOpsEntityProvider.fromConfig(env.config, {
+ logger: env.logger,
+ // optional: alternatively, use scheduler with schedule defined in app-config.yaml
+ schedule: env.scheduler.createScheduledTaskRunner({
+ frequency: Duration.fromObject({ minutes: 30 }),
+ timeout: Duration.fromObject({ minutes: 3 }),
+ frequency: { minutes: 30 },
+ timeout: { minutes: 3 },
+ }),
+ // optional: alternatively, use schedule
+ scheduler: env.scheduler,
+ }),
+);
```
docs/integrations/bitbucketServer/discovery.md
+25 -11
View File
@@ -37,10 +37,13 @@ And then add the entity provider to your catalog builder:
+ builder.addEntityProvider(
+ BitbucketServerEntityProvider.fromConfig(env.config, {
+ logger: env.logger,
+ // optional: alternatively, use scheduler with schedule defined in app-config.yaml
+ schedule: env.scheduler.createScheduledTaskRunner({
+ frequency: { minutes: 30 },
+ timeout: { minutes: 3 },
+ }),
+ // optional: alternatively, use schedule
+ scheduler: env.scheduler,
+ }),
+ );
@@ -66,19 +69,33 @@ catalog:
filters: # optional
projectKey: '^apis-.*$' # optional; RegExp
repoSlug: '^service-.*$' # optional; RegExp
schedule: # optional; same options as in TaskScheduleDefinition
# supports cron, ISO duration, "human duration" as used in code
frequency: { minutes: 30 }
# supports ISO duration, "human duration" as used in code
timeout: { minutes: 3 }
```
- **host**:
- **`host`**:
The host of the Bitbucket Server instance, **note**: the host needs to registered as an integration as well, see [location](locations.md).
- **`catalogPath`** _(optional)_:
Default: `/catalog-info.yaml`.
Path where to look for `catalog-info.yaml` files.
When started with `/`, it is an absolute path from the repo root.
- **filters** _(optional)_:
- **`filters`** _(optional)_:
- **`projectKey`** _(optional)_:
Regular expression used to filter results based on the project key.
- **repoSlug** _(optional)_:
- **`repoSlug`** _(optional)_:
Regular expression used to filter results based on the repo slug.
- **`schedule`** _(optional)_:
- **`frequency`**:
How often you want the task to run. The system does its best to avoid overlapping invocations.
- **`timeout`**:
The maximum amount of time that a single task invocation can take.
- **`initialDelay`** _(optional)_:
The amount of time that should pass before the first invocation happens.
- **`scope`** _(optional)_:
`'global'` or `'local'`. Sets the scope of concurrency control.
## Custom location processing
@@ -93,18 +110,15 @@ repository.
```typescript
const provider = BitbucketServerEntityProvider.fromConfig(env.config, {
logger: env.logger,
schedule: env.scheduler.createScheduledTaskRunner({
frequency: { minutes: 30 },
timeout: { minutes: 3 },
}),
schedule: env.scheduler,
parser: async function* customLocationParser(options: {
location: LocationSpec,
client: BitbucketServerClient,
location: LocationSpec;
client: BitbucketServerClient;
}) {
// Custom logic for interpreting the matching repository
// See defaultBitbucketServerLocationParser for an example
}
);
},
});
```
## Alternative
docs/integrations/gerrit/discovery.md
+12 -4
View File
@@ -32,10 +32,13 @@ const builder = await CatalogBuilder.create(env);
builder.addEntityProvider(
GerritEntityProvider.fromConfig(env.config, {
logger: env.logger,
// optional: alternatively, use scheduler with schedule defined in app-config.yaml
schedule: env.scheduler.createScheduledTaskRunner({
frequency: { minutes: 30 },
timeout: { minutes: 3 },
}),
// optional: alternatively, use schedule
scheduler: env.scheduler,
}),
);
```
@@ -54,6 +57,11 @@ catalog:
host: gerrit-your-company.com
branch: master # Optional
query: 'state=ACTIVE&prefix=webapps'
schedule: # optional; same options as in TaskScheduleDefinition
# supports cron, ISO duration, "human duration" as used in code
frequency: { minutes: 30 }
# supports ISO duration, "human duration" as used in code
timeout: { minutes: 3 }
backend:
host: gerrit-your-company.com
branch: master # Optional
@@ -62,8 +70,8 @@ catalog:
The provider configuration is composed of three parts:
- host, the host of the Gerrit integration to use.
- branch, the branch where we will look for catalog entities (defaults to "master").
- query, this string is directly used as the argument to the "List Project" API.
Typically you will want to have some filter here to exclude projects that will
- **`host`**: the host of the Gerrit integration to use.
- **`branch`** _(optional)_: the branch where we will look for catalog entities (defaults to "master").
- **`query`**: this string is directly used as the argument to the "List Project" API.
Typically, you will want to have some filter here to exclude projects that will
never contain any catalog files.
docs/integrations/gitlab/discovery.md
+8
View File
@@ -25,6 +25,11 @@ catalog:
group: example-group # Optional. Group and subgroup (if needed) to look for repositories. If not present the whole project will be scanned
entityFilename: catalog-info.yaml # Optional. Defaults to `catalog-info.yaml`
projectPattern: /[\s\S]*/ # Optional. Filters found projects based on provided patter. Defaults to `/[\s\S]*/`, what means to not filter anything
schedule: # optional; same options as in TaskScheduleDefinition
# supports cron, ISO duration, "human duration" as used in code
frequency: { minutes: 30 }
# supports ISO duration, "human duration" as used in code
timeout: { minutes: 3 }
```
As this provider is not one of the default providers, you will first need to install
@@ -47,10 +52,13 @@ const builder = await CatalogBuilder.create(env);
builder.addEntityProvider(
...GitlabDiscoveryEntityProvider.fromConfig(env.config, {
logger: env.logger,
// optional: alternatively, use scheduler with schedule defined in app-config.yaml
schedule: env.scheduler.createScheduledTaskRunner({
frequency: { minutes: 30 },
timeout: { minutes: 3 },
}),
// optional: alternatively, use schedule
scheduler: env.scheduler,
}),
);
```
plugins/catalog-backend-module-aws/api-report.md
+9 -1
View File
@@ -3,6 +3,7 @@
> Do not edit this file. It is a report generated by [API Extractor](https://api-extractor.com/).
```ts
import { BackendFeature } from '@backstage/backend-plugin-api';
import { CatalogProcessor } from '@backstage/plugin-catalog-backend';
import { CatalogProcessorEmit } from '@backstage/plugin-catalog-backend';
import { CatalogProcessorParser } from '@backstage/plugin-catalog-backend';
@@ -12,6 +13,7 @@ import { EntityProvider } from '@backstage/plugin-catalog-backend';
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
import { LocationSpec } from '@backstage/plugin-catalog-backend';
import { Logger } from 'winston';
import { PluginTaskScheduler } from '@backstage/backend-tasks';
import { TaskRunner } from '@backstage/backend-tasks';
import { UrlReader } from '@backstage/backend-common';
@@ -77,7 +79,8 @@ export class AwsS3EntityProvider implements EntityProvider {
configRoot: Config,
options: {
logger: Logger;
schedule: TaskRunner;
schedule?: TaskRunner;
scheduler?: PluginTaskScheduler;
},
): AwsS3EntityProvider[];
// (undocumented)
@@ -85,4 +88,9 @@ export class AwsS3EntityProvider implements EntityProvider {
// (undocumented)
refresh(logger: Logger): Promise<void>;
}
// @alpha
export const awsS3EntityProviderCatalogModule: (
options?: undefined,
) => BackendFeature;
```
plugins/catalog-backend-module-aws/config.d.ts
Vendored
+10 -9
View File
@@ -14,11 +14,10 @@
* limitations under the License.
*/
import { TaskScheduleDefinitionConfig } from '@backstage/backend-tasks';
export interface Config {
catalog?: {
/**
* List of processor-specific options and attributes
*/
processors?: {
/**
* AwsOrganizationCloudAccountProcessor configuration
@@ -32,9 +31,6 @@ export interface Config {
};
};
};
/**
* List of provider-specific options and attributes
*/
providers?: {
/**
* AwsS3EntityProvider configuration
@@ -45,22 +41,23 @@ export interface Config {
| {
/**
* (Required) AWS S3 Bucket Name
* @visibility backend
*/
bucketName: string;
/**
* (Optional) AWS S3 Object key prefix
* If not set, all keys will be accepted, no filtering will be applied.
* @visibility backend
*/
prefix?: string;
/**
* (Optional) AWS Region.
* If not set, AWS_REGION environment variable or aws config file will be used.
* @see https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/setting-region.html
* @visibility backend
*/
region?: string;
/**
* (Optional) TaskScheduleDefinition for the refresh.
*/
schedule?: TaskScheduleDefinitionConfig;
}
| Record<
string,
@@ -83,6 +80,10 @@ export interface Config {
* @visibility backend
*/
region?: string;
/**
* (Optional) TaskScheduleDefinition for the refresh.
*/
schedule?: TaskScheduleDefinitionConfig;
}
>;
};
plugins/catalog-backend-module-aws/package.json
+11 -5
View File
@@ -7,6 +7,7 @@
"license": "Apache-2.0",
"publishConfig": {
"access": "public",
"alphaTypes": "dist/index.alpha.d.ts",
"main": "dist/index.cjs.js",
"types": "dist/index.d.ts"
},
@@ -23,22 +24,24 @@
"backstage"
],
"scripts": {
"build": "backstage-cli package build",
"start": "backstage-cli package start",
"build": "backstage-cli package build --experimental-type-build",
"lint": "backstage-cli package lint",
"test": "backstage-cli package test",
"prepack": "backstage-cli package prepack",
"postpack": "backstage-cli package postpack",
"clean": "backstage-cli package clean",
"start": "backstage-cli package start"
"clean": "backstage-cli package clean"
},
"dependencies": {
"@backstage/backend-common": "workspace:^",
"@backstage/backend-plugin-api": "workspace:^",
"@backstage/backend-tasks": "workspace:^",
"@backstage/catalog-model": "workspace:^",
"@backstage/config": "workspace:^",
"@backstage/errors": "workspace:^",
"@backstage/integration": "workspace:^",
"@backstage/plugin-catalog-backend": "workspace:^",
"@backstage/plugin-catalog-node": "workspace:^",
"@backstage/types": "workspace:^",
"aws-sdk": "^2.840.0",
"lodash": "^4.17.21",
@@ -47,14 +50,17 @@
"winston": "^3.2.1"
},
"devDependencies": {
"@backstage/backend-test-utils": "workspace:^",
"@backstage/cli": "workspace:^",
"@types/lodash": "^4.14.151",
"aws-sdk-mock": "^5.2.1",
"luxon": "^3.0.0",
"yaml": "^2.0.0"
},
"files": [
"dist",
"config.d.ts"
"alpha",
"config.d.ts",
"dist"
],
"configSchema": "config.d.ts"
}
plugins/catalog-backend-module-aws/src/index.ts
+1
View File
@@ -22,4 +22,5 @@
export * from './processors';
export * from './providers';
export { awsS3EntityProviderCatalogModule } from './service/AwsS3EntityProviderCatalogModule';
export * from './types';
plugins/catalog-backend-module-aws/src/providers/AwsS3EntityProvider.test.ts
+94 -2
View File
@@ -15,7 +15,11 @@
*/
import { getVoidLogger } from '@backstage/backend-common';
import { TaskInvocationDefinition, TaskRunner } from '@backstage/backend-tasks';
import {
PluginTaskScheduler,
TaskInvocationDefinition,
TaskRunner,
} from '@backstage/backend-tasks';
import { ConfigReader } from '@backstage/config';
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
import { AwsS3EntityProvider } from './AwsS3EntityProvider';
@@ -83,6 +87,7 @@ describe('AwsS3EntityProvider', () => {
expectedBaseUrl: string,
names: Record<string, string>,
integrationConfig?: object,
scheduleInConfig?: boolean,
) => {
const config = new ConfigReader({
integrations: {
@@ -97,15 +102,25 @@ describe('AwsS3EntityProvider', () => {
},
});
const schedulingConfig: Record<string, any> = {};
const schedule = new PersistingTaskRunner();
const entityProviderConnection: EntityProviderConnection = {
applyMutation: jest.fn(),
refresh: jest.fn(),
};
if (scheduleInConfig) {
schedulingConfig.scheduler = {
createScheduledTaskRunner: (_: any) => schedule,
} as unknown as PluginTaskScheduler;
} else {
schedulingConfig.schedule = schedule;
}
const provider = AwsS3EntityProvider.fromConfig(config, {
...schedulingConfig,
logger,
schedule,
})[0];
expect(provider.getProviderName()).toEqual(`awsS3-provider:${providerId}`);
@@ -224,4 +239,81 @@ describe('AwsS3EntityProvider', () => {
},
);
});
it('fail without schedule and scheduler', () => {
const config = new ConfigReader({
catalog: {
providers: {
awsS3: {
test: {
bucketName: 'bucket-1',
prefix: 'sub/dir/',
region: 'eu-west-1',
},
},
},
},
});
expect(() =>
AwsS3EntityProvider.fromConfig(config, {
logger,
}),
).toThrow('Either schedule or scheduler must be provided');
});
it('fail with scheduler but no schedule config', () => {
const scheduler = {
createScheduledTaskRunner: (_: any) => jest.fn(),
} as unknown as PluginTaskScheduler;
const config = new ConfigReader({
catalog: {
providers: {
awsS3: {
test: {
bucketName: 'bucket-1',
prefix: 'sub/dir/',
region: 'eu-west-1',
},
},
},
},
});
expect(() =>
AwsS3EntityProvider.fromConfig(config, {
logger,
scheduler,
}),
).toThrow(
'No schedule provided neither via code nor config for awsS3-provider:test',
);
});
// eslint-disable-next-line jest/expect-expect
it('single simple provider config with schedule in config', async () => {
return expectMutation(
'regionalStatic',
{
bucketName: 'bucket-1',
prefix: 'sub/dir/',
region: 'eu-west-1',
schedule: {
frequency: 'PT30M',
timeout: {
minutes: 3,
},
},
},
'https://s3.eu-west-1.amazonaws.com/bucket-1/sub/dir/',
{
'key1.yaml': 'generated-7f6d5861b0b3401a38b5fe62e6c7ca11da5fd6d8',
'key2.yaml': 'generated-a290be145586042af7d80715626399c9d661718d',
'key3.yaml': 'generated-8d75f78ed9fa618ce433b226dc24eeab441f3a2d',
'key 4.yaml': 'generated-1e0249dcb5805fc2ce6ac2d3c4d2a3ef4f1270c0',
},
undefined,
true,
);
});
});
plugins/catalog-backend-module-aws/src/providers/AwsS3EntityProvider.ts
+29 -15
View File
@@ -14,7 +14,7 @@
* limitations under the License.
*/
import { TaskRunner } from '@backstage/backend-tasks';
import { PluginTaskScheduler, TaskRunner } from '@backstage/backend-tasks';
import { Config } from '@backstage/config';
import { AwsS3Integration, ScmIntegrations } from '@backstage/integration';
import {
@@ -49,7 +49,8 @@ export class AwsS3EntityProvider implements EntityProvider {
configRoot: Config,
options: {
logger: Logger;
schedule: TaskRunner;
schedule?: TaskRunner;
scheduler?: PluginTaskScheduler;
},
): AwsS3EntityProvider[] {
const providerConfigs = readAwsS3Configs(configRoot);
@@ -67,22 +68,35 @@ export class AwsS3EntityProvider implements EntityProvider {
throw new Error('No integration found for awsS3');
}
return providerConfigs.map(
providerConfig =>
new AwsS3EntityProvider(
providerConfig,
integration,
options.logger,
options.schedule,
),
);
if (!options.schedule && !options.scheduler) {
throw new Error('Either schedule or scheduler must be provided.');
}
return providerConfigs.map(providerConfig => {
if (!options.schedule && !providerConfig.schedule) {
throw new Error(
`No schedule provided neither via code nor config for awsS3-provider:${providerConfig.id}.`,
);
}
const taskRunner =
options.schedule ??
options.scheduler!.createScheduledTaskRunner(providerConfig.schedule!);
return new AwsS3EntityProvider(
providerConfig,
integration,
options.logger,
taskRunner,
);
});
}
private constructor(
private readonly config: AwsS3Config,
integration: AwsS3Integration,
logger: Logger,
schedule: TaskRunner,
taskRunner: TaskRunner,
) {
this.logger = logger.child({
target: this.getProviderName(),
@@ -99,13 +113,13 @@ export class AwsS3EntityProvider implements EntityProvider {
s3ForcePathStyle: integration.config.s3ForcePathStyle,
});
this.scheduleFn = this.createScheduleFn(schedule);
this.scheduleFn = this.createScheduleFn(taskRunner);
}
private createScheduleFn(schedule: TaskRunner): () => Promise<void> {
private createScheduleFn(taskRunner: TaskRunner): () => Promise<void> {
return async () => {
const taskId = `${this.getProviderName()}:refresh`;
return schedule.run({
return taskRunner.run({
id: taskId,
fn: async () => {
const logger = this.logger.child({
plugins/catalog-backend-module-aws/src/providers/config.test.ts
+20 -2
View File
@@ -15,6 +15,7 @@
*/
import { ConfigReader } from '@backstage/config';
import { Duration } from 'luxon';
import { readAwsS3Configs } from './config';
describe('readAwsS3Configs', () => {
@@ -54,17 +55,26 @@ describe('readAwsS3Configs', () => {
const provider3 = {
bucketName: 'bucket-3',
};
const provider4 = {
bucketName: 'bucket-4',
schedule: {
frequency: 'PT30M',
timeout: {
minutes: 3,
},
},
};
const config = {
catalog: {
providers: {
awsS3: { provider1, provider2, provider3 },
awsS3: { provider1, provider2, provider3, provider4 },
},
},
};
const actual = readAwsS3Configs(new ConfigReader(config));
expect(actual).toHaveLength(3);
expect(actual).toHaveLength(4);
expect(actual[0]).toEqual({
...provider1,
id: 'provider1',
@@ -77,6 +87,14 @@ describe('readAwsS3Configs', () => {
...provider3,
id: 'provider3',
});
expect(actual[3]).toEqual({
...provider4,
id: 'provider4',
schedule: {
...provider4.schedule,
frequency: Duration.fromISO(provider4.schedule.frequency),
},
});
});
it('fails if bucketName is missing', () => {
plugins/catalog-backend-module-aws/src/providers/config.ts
+6
View File
@@ -14,6 +14,7 @@
* limitations under the License.
*/
import { readTaskScheduleDefinitionFromConfig } from '@backstage/backend-tasks';
import { Config } from '@backstage/config';
import { AwsS3Config } from './types';
@@ -46,10 +47,15 @@ function readAwsS3Config(id: string, config: Config): AwsS3Config {
const region = config.getOptionalString('region');
const prefix = config.getOptionalString('prefix');
const schedule = config.has('schedule')
? readTaskScheduleDefinitionFromConfig(config.getConfig('schedule'))
: undefined;
return {
id,
bucketName,
region,
prefix,
schedule,
};
}
plugins/catalog-backend-module-aws/src/providers/types.ts
+3
View File
@@ -14,9 +14,12 @@
* limitations under the License.
*/
import { TaskScheduleDefinition } from '@backstage/backend-tasks';
export type AwsS3Config = {
id: string;
bucketName: string;
prefix?: string;
region?: string;
schedule?: TaskScheduleDefinition;
};
plugins/catalog-backend-module-aws/src/service/AwsS3EntityProviderCatalogModule.test.ts
+84
View File
@@ -0,0 +1,84 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { getVoidLogger } from '@backstage/backend-common';
import {
configServiceRef,
loggerServiceRef,
schedulerServiceRef,
} from '@backstage/backend-plugin-api';
import {
PluginTaskScheduler,
TaskScheduleDefinition,
} from '@backstage/backend-tasks';
import { startTestBackend } from '@backstage/backend-test-utils';
import { ConfigReader } from '@backstage/config';
import { catalogProcessingExtensionPoint } from '@backstage/plugin-catalog-node';
import { Duration } from 'luxon';
import { awsS3EntityProviderCatalogModule } from './AwsS3EntityProviderCatalogModule';
import { AwsS3EntityProvider } from '../providers';
describe('awsS3EntityProviderCatalogModule', () => {
it('should register provider at the catalog extension point', async () => {
let addedProviders: Array<AwsS3EntityProvider> | undefined;
let usedSchedule: TaskScheduleDefinition | undefined;
const extensionPoint = {
addEntityProvider: (providers: any) => {
addedProviders = providers;
},
};
const runner = jest.fn();
const scheduler = {
createScheduledTaskRunner: (schedule: TaskScheduleDefinition) => {
usedSchedule = schedule;
return runner;
},
} as unknown as PluginTaskScheduler;
const config = new ConfigReader({
catalog: {
providers: {
awsS3: {
bucketName: 'module-test',
schedule: {
frequency: 'P1M',
timeout: 'PT3M',
},
},
},
},
});
await startTestBackend({
extensionPoints: [[catalogProcessingExtensionPoint, extensionPoint]],
services: [
[configServiceRef, config],
[loggerServiceRef, getVoidLogger()],
[schedulerServiceRef, scheduler],
],
features: [awsS3EntityProviderCatalogModule()],
});
expect(usedSchedule?.frequency).toEqual(Duration.fromISO('P1M'));
expect(usedSchedule?.timeout).toEqual(Duration.fromISO('PT3M'));
expect(addedProviders?.length).toEqual(1);
expect(addedProviders?.pop()?.getProviderName()).toEqual(
'awsS3-provider:default',
);
expect(runner).not.toHaveBeenCalled();
});
});
plugins/catalog-backend-module-aws/src/service/AwsS3EntityProviderCatalogModule.ts
+53
View File
@@ -0,0 +1,53 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import {
createBackendModule,
loggerToWinstonLogger,
configServiceRef,
loggerServiceRef,
schedulerServiceRef,
} from '@backstage/backend-plugin-api';
import { catalogProcessingExtensionPoint } from '@backstage/plugin-catalog-node';
import { AwsS3EntityProvider } from '../providers';
/**
* Registers the AwsS3EntityProvider with the catalog processing extension point.
*
* @alpha
*/
export const awsS3EntityProviderCatalogModule = createBackendModule({
pluginId: 'catalog',
moduleId: 'awsS3EntityProvider',
register(env) {
env.registerInit({
deps: {
config: configServiceRef,
catalog: catalogProcessingExtensionPoint,
logger: loggerServiceRef,
scheduler: schedulerServiceRef,
},
async init({ config, catalog, logger, scheduler }) {
catalog.addEntityProvider(
AwsS3EntityProvider.fromConfig(config, {
logger: loggerToWinstonLogger(logger),
scheduler,
}),
);
},
});
},
});
plugins/catalog-backend-module-azure/api-report.md
+9 -1
View File
@@ -3,6 +3,7 @@
> Do not edit this file. It is a report generated by [API Extractor](https://api-extractor.com/).
```ts
import { BackendFeature } from '@backstage/backend-plugin-api';
import { CatalogProcessor } from '@backstage/plugin-catalog-backend';
import { CatalogProcessorEmit } from '@backstage/plugin-catalog-backend';
import { Config } from '@backstage/config';
@@ -10,6 +11,7 @@ import { EntityProvider } from '@backstage/plugin-catalog-backend';
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
import { LocationSpec } from '@backstage/plugin-catalog-backend';
import { Logger } from 'winston';
import { PluginTaskScheduler } from '@backstage/backend-tasks';
import { ScmIntegrationRegistry } from '@backstage/integration';
import { TaskRunner } from '@backstage/backend-tasks';
@@ -45,7 +47,8 @@ export class AzureDevOpsEntityProvider implements EntityProvider {
configRoot: Config,
options: {
logger: Logger;
schedule: TaskRunner;
schedule?: TaskRunner;
scheduler?: PluginTaskScheduler;
},
): AzureDevOpsEntityProvider[];
// (undocumented)
@@ -53,4 +56,9 @@ export class AzureDevOpsEntityProvider implements EntityProvider {
// (undocumented)
refresh(logger: Logger): Promise<void>;
}
// @alpha
export const azureDevOpsEntityProviderCatalogModule: (
options?: undefined,
) => BackendFeature;
```
plugins/catalog-backend-module-azure/config.d.ts
Vendored
+6 -5
View File
@@ -14,34 +14,35 @@
* limitations under the License.
*/
import { TaskScheduleDefinitionConfig } from '@backstage/backend-tasks';
interface AzureDevOpsConfig {
/**
* (Optional) The DevOps host; leave empty for `dev.azure.com`, otherwise set to your self-hosted instance host.
* @visibility backend
*/
host: string;
/**
* (Required) Your organization slug.
* @visibility backend
*/
organization: string;
/**
* (Required) Your project slug.
* @visibility backend
*/
project: string;
/**
* (Optional) The repository name. Wildcards are supported as show on the examples above.
* If not set, all repositories will be searched.
* @visibility backend
*/
repository?: string;
/**
* (Optional) Where to find catalog-info.yaml files. Wildcards are supported.
* If not set, defaults to /catalog-info.yaml.
* @visibility backend
*/
path?: string;
/**
* (Optional) TaskScheduleDefinition for the refresh.
*/
schedule?: TaskScheduleDefinitionConfig;
}
export interface Config {
plugins/catalog-backend-module-azure/package.json
+11 -6
View File
@@ -7,6 +7,7 @@
"license": "Apache-2.0",
"publishConfig": {
"access": "public",
"alphaTypes": "dist/index.alpha.d.ts",
"main": "dist/index.cjs.js",
"types": "dist/index.d.ts"
},
@@ -23,22 +24,24 @@
"backstage"
],
"scripts": {
"build": "backstage-cli package build",
"start": "backstage-cli package start",
"build": "backstage-cli package build --experimental-type-build",
"lint": "backstage-cli package lint",
"test": "backstage-cli package test",
"prepack": "backstage-cli package prepack",
"postpack": "backstage-cli package postpack",
"clean": "backstage-cli package clean",
"start": "backstage-cli package start"
"clean": "backstage-cli package clean"
},
"dependencies": {
"@backstage/backend-common": "workspace:^",
"@backstage/backend-plugin-api": "workspace:^",
"@backstage/backend-tasks": "workspace:^",
"@backstage/catalog-model": "workspace:^",
"@backstage/config": "workspace:^",
"@backstage/errors": "workspace:^",
"@backstage/integration": "workspace:^",
"@backstage/plugin-catalog-backend": "workspace:^",
"@backstage/plugin-catalog-node": "workspace:^",
"@backstage/types": "workspace:^",
"lodash": "^4.17.21",
"msw": "^0.47.0",
@@ -49,11 +52,13 @@
"devDependencies": {
"@backstage/backend-test-utils": "workspace:^",
"@backstage/cli": "workspace:^",
"@types/lodash": "^4.14.151"
"@types/lodash": "^4.14.151",
"luxon": "^3.0.0"
},
"files": [
"dist",
"config.d.ts"
"alpha",
"config.d.ts",
"dist"
],
"configSchema": "config.d.ts"
}
plugins/catalog-backend-module-azure/src/index.ts
+1
View File
@@ -22,3 +22,4 @@
export { AzureDevOpsDiscoveryProcessor } from './processors';
export { AzureDevOpsEntityProvider } from './providers';
export { azureDevOpsEntityProviderCatalogModule } from './service/AzureDevOpsEntityProviderCatalogModule';
plugins/catalog-backend-module-azure/src/providers/AzureDevOpsEntityProvider.test.ts
+104 -2
View File
@@ -15,7 +15,11 @@
*/
import { getVoidLogger } from '@backstage/backend-common';
import { TaskInvocationDefinition, TaskRunner } from '@backstage/backend-tasks';
import {
PluginTaskScheduler,
TaskInvocationDefinition,
TaskRunner,
} from '@backstage/backend-tasks';
import { ConfigReader } from '@backstage/config';
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
import { CodeSearchResultItem } from '../lib';
@@ -52,6 +56,7 @@ describe('AzureDevOpsEntityProvider', () => {
expectedBaseUrl: string,
names: Record<string, string>,
integrationConfig?: object,
scheduleInConfig?: boolean,
) => {
const config = new ConfigReader({
integrations: {
@@ -74,9 +79,18 @@ describe('AzureDevOpsEntityProvider', () => {
refresh: jest.fn(),
};
const schedulingConfig: Record<string, any> = {};
if (scheduleInConfig) {
schedulingConfig.scheduler = {
createScheduledTaskRunner: (_: any) => schedule,
} as unknown as PluginTaskScheduler;
} else {
schedulingConfig.schedule = schedule;
}
const provider = AzureDevOpsEntityProvider.fromConfig(config, {
...schedulingConfig,
logger,
schedule,
})[0];
expect(provider.getProviderName()).toEqual(
`AzureDevOpsEntityProvider:${providerId}`,
@@ -193,4 +207,92 @@ describe('AzureDevOpsEntityProvider', () => {
},
);
});
it('fail without schedule and scheduler', () => {
const config = new ConfigReader({
catalog: {
providers: {
azureDevOps: {
test: {
organization: 'myorganization',
project: 'myproject',
},
},
},
},
});
expect(() =>
AzureDevOpsEntityProvider.fromConfig(config, {
logger,
}),
).toThrow('Either schedule or scheduler must be provided');
});
it('fail with scheduler but no schedule config', () => {
const scheduler = {
createScheduledTaskRunner: (_: any) => jest.fn(),
} as unknown as PluginTaskScheduler;
const config = new ConfigReader({
catalog: {
providers: {
azureDevOps: {
test: {
organization: 'myorganization',
project: 'myproject',
},
},
},
},
});
expect(() =>
AzureDevOpsEntityProvider.fromConfig(config, {
logger,
scheduler,
}),
).toThrow(
'No schedule provided neither via code nor config for AzureDevOpsEntityProvider:test',
);
});
// eslint-disable-next-line jest/expect-expect
it('single simple provider config with schedule in config', async () => {
return expectMutation(
'allReposMultipleFiles',
{
organization: 'myorganization',
project: 'myproject',
schedule: {
frequency: 'PT30M',
timeout: {
minutes: 3,
},
},
},
[
{
fileName: 'catalog-info.yaml',
path: '/catalog-info.yaml',
repository: {
name: 'myrepo',
},
},
{
fileName: 'catalog-info.yaml',
path: '/catalog-info.yaml',
repository: {
name: 'myotherrepo',
},
},
],
'https://dev.azure.com/myorganization/myproject',
{
'myrepo?path=/catalog-info.yaml':
'generated-87865246726bb12a8c4fb4f914443f1fbb91648c',
'myotherrepo?path=/catalog-info.yaml':
'generated-2deccac384c34d0dca37be0ebb4b1c8cf6913fe1',
},
);
});
});
plugins/catalog-backend-module-azure/src/providers/AzureDevOpsEntityProvider.ts
+22 -7
View File
@@ -14,7 +14,7 @@
* limitations under the License.
*/
import { TaskRunner } from '@backstage/backend-tasks';
import { PluginTaskScheduler, TaskRunner } from '@backstage/backend-tasks';
import { Config } from '@backstage/config';
import { AzureIntegration, ScmIntegrations } from '@backstage/integration';
import {
@@ -45,11 +45,16 @@ export class AzureDevOpsEntityProvider implements EntityProvider {
configRoot: Config,
options: {
logger: Logger;
schedule: TaskRunner;
schedule?: TaskRunner;
scheduler?: PluginTaskScheduler;
},
): AzureDevOpsEntityProvider[] {
const providerConfigs = readAzureDevOpsConfigs(configRoot);
if (!options.schedule && !options.scheduler) {
throw new Error('Either schedule or scheduler must be provided.');
}
return providerConfigs.map(providerConfig => {
const integration = ScmIntegrations.fromConfig(configRoot).azure.byHost(
providerConfig.host,
@@ -61,11 +66,21 @@ export class AzureDevOpsEntityProvider implements EntityProvider {
);
}
if (!options.schedule && !providerConfig.schedule) {
throw new Error(
`No schedule provided neither via code nor config for AzureDevOpsEntityProvider:${providerConfig.id}.`,
);
}
const taskRunner =
options.schedule ??
options.scheduler!.createScheduledTaskRunner(providerConfig.schedule!);
return new AzureDevOpsEntityProvider(
providerConfig,
integration,
options.logger,
options.schedule,
taskRunner,
);
});
}
@@ -74,19 +89,19 @@ export class AzureDevOpsEntityProvider implements EntityProvider {
private readonly config: AzureDevOpsConfig,
private readonly integration: AzureIntegration,
logger: Logger,
schedule: TaskRunner,
taskRunner: TaskRunner,
) {
this.logger = logger.child({
target: this.getProviderName(),
});
this.scheduleFn = this.createScheduleFn(schedule);
this.scheduleFn = this.createScheduleFn(taskRunner);
}
private createScheduleFn(schedule: TaskRunner): () => Promise<void> {
private createScheduleFn(taskRunner: TaskRunner): () => Promise<void> {
return async () => {
const taskId = `${this.getProviderName()}:refresh`;
return schedule.run({
return taskRunner.run({
id: taskId,
fn: async () => {
const logger = this.logger.child({
plugins/catalog-backend-module-azure/src/providers/config.test.ts
+24 -2
View File
@@ -15,6 +15,7 @@
*/
import { ConfigReader } from '@backstage/config';
import { Duration } from 'luxon';
import { readAzureDevOpsConfigs } from './config';
describe('readAzureDevOpsConfigs', () => {
@@ -33,17 +34,27 @@ describe('readAzureDevOpsConfigs', () => {
project: 'myproject',
repository: 'service-*',
};
const provider4 = {
organization: 'mycompany',
project: 'myproject',
schedule: {
frequency: 'PT30M',
timeout: {
minutes: 3,
},
},
};
const config = {
catalog: {
providers: {
azureDevOps: { provider1, provider2, provider3 },
azureDevOps: { provider1, provider2, provider3, provider4 },
},
},
};
const actual = readAzureDevOpsConfigs(new ConfigReader(config));
expect(actual).toHaveLength(3);
expect(actual).toHaveLength(4);
expect(actual[0]).toEqual({
...provider1,
path: '/catalog-info.yaml',
@@ -63,5 +74,16 @@ describe('readAzureDevOpsConfigs', () => {
path: '/catalog-info.yaml',
id: 'provider3',
});
expect(actual[3]).toEqual({
...provider4,
host: 'dev.azure.com',
path: '/catalog-info.yaml',
repository: '*',
id: 'provider4',
schedule: {
...provider4.schedule,
frequency: Duration.fromISO(provider4.schedule.frequency),
},
});
});
});
plugins/catalog-backend-module-azure/src/providers/config.ts
+6
View File
@@ -14,6 +14,7 @@
* limitations under the License.
*/
import { readTaskScheduleDefinitionFromConfig } from '@backstage/backend-tasks';
import { Config } from '@backstage/config';
import { AzureDevOpsConfig } from './types';
@@ -42,6 +43,10 @@ function readAzureDevOpsConfig(id: string, config: Config): AzureDevOpsConfig {
const repository = config.getOptionalString('repository') || '*';
const path = config.getOptionalString('path') || '/catalog-info.yaml';
const schedule = config.has('schedule')
? readTaskScheduleDefinitionFromConfig(config.getConfig('schedule'))
: undefined;
return {
id,
host,
@@ -49,5 +54,6 @@ function readAzureDevOpsConfig(id: string, config: Config): AzureDevOpsConfig {
project,
repository,
path,
schedule,
};
}
plugins/catalog-backend-module-azure/src/providers/types.ts
+3
View File
@@ -14,6 +14,8 @@
* limitations under the License.
*/
import { TaskScheduleDefinition } from '@backstage/backend-tasks';
export type AzureDevOpsConfig = {
id: string;
host: string;
@@ -21,4 +23,5 @@ export type AzureDevOpsConfig = {
project: string;
repository: string;
path: string;
schedule?: TaskScheduleDefinition;
};
plugins/catalog-backend-module-azure/src/service/AzureDevOpsEntityProviderCatalogModule.test.ts
+87
View File
@@ -0,0 +1,87 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { getVoidLogger } from '@backstage/backend-common';
import {
configServiceRef,
loggerServiceRef,
schedulerServiceRef,
} from '@backstage/backend-plugin-api';
import {
PluginTaskScheduler,
TaskScheduleDefinition,
} from '@backstage/backend-tasks';
import { startTestBackend } from '@backstage/backend-test-utils';
import { ConfigReader } from '@backstage/config';
import { catalogProcessingExtensionPoint } from '@backstage/plugin-catalog-node';
import { Duration } from 'luxon';
import { azureDevOpsEntityProviderCatalogModule } from './AzureDevOpsEntityProviderCatalogModule';
import { AzureDevOpsEntityProvider } from '../providers';
describe('azureDevOpsEntityProviderCatalogModule', () => {
it('should register provider at the catalog extension point', async () => {
let addedProviders: Array<AzureDevOpsEntityProvider> | undefined;
let usedSchedule: TaskScheduleDefinition | undefined;
const extensionPoint = {
addEntityProvider: (providers: any) => {
addedProviders = providers;
},
};
const runner = jest.fn();
const scheduler = {
createScheduledTaskRunner: (schedule: TaskScheduleDefinition) => {
usedSchedule = schedule;
return runner;
},
} as unknown as PluginTaskScheduler;
const config = new ConfigReader({
catalog: {
providers: {
azureDevOps: {
test: {
organization: 'myorganization',
project: 'myproject',
schedule: {
frequency: 'P1M',
timeout: 'PT3M',
},
},
},
},
},
});
await startTestBackend({
extensionPoints: [[catalogProcessingExtensionPoint, extensionPoint]],
services: [
[configServiceRef, config],
[loggerServiceRef, getVoidLogger()],
[schedulerServiceRef, scheduler],
],
features: [azureDevOpsEntityProviderCatalogModule()],
});
expect(usedSchedule?.frequency).toEqual(Duration.fromISO('P1M'));
expect(usedSchedule?.timeout).toEqual(Duration.fromISO('PT3M'));
expect(addedProviders?.length).toEqual(1);
expect(addedProviders?.pop()?.getProviderName()).toEqual(
'AzureDevOpsEntityProvider:test',
);
expect(runner).not.toHaveBeenCalled();
});
});
plugins/catalog-backend-module-azure/src/service/AzureDevOpsEntityProviderCatalogModule.ts
+53
View File
@@ -0,0 +1,53 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import {
createBackendModule,
loggerToWinstonLogger,
configServiceRef,
loggerServiceRef,
schedulerServiceRef,
} from '@backstage/backend-plugin-api';
import { catalogProcessingExtensionPoint } from '@backstage/plugin-catalog-node';
import { AzureDevOpsEntityProvider } from '../providers';
/**
* Registers the AzureDevOpsEntityProvider with the catalog processing extension point.
*
* @alpha
*/
export const azureDevOpsEntityProviderCatalogModule = createBackendModule({
pluginId: 'catalog',
moduleId: 'azureDevOpsEntityProvider',
register(env) {
env.registerInit({
deps: {
config: configServiceRef,
catalog: catalogProcessingExtensionPoint,
logger: loggerServiceRef,
scheduler: schedulerServiceRef,
},
async init({ config, catalog, logger, scheduler }) {
catalog.addEntityProvider(
AzureDevOpsEntityProvider.fromConfig(config, {
logger: loggerToWinstonLogger(logger),
scheduler,
}),
);
},
});
},
});
plugins/catalog-backend-module-bitbucket-server/api-report.md
+9 -1
View File
@@ -3,6 +3,7 @@
> Do not edit this file. It is a report generated by [API Extractor](https://api-extractor.com/).
```ts
import { BackendFeature } from '@backstage/backend-plugin-api';
import { BitbucketServerIntegrationConfig } from '@backstage/integration';
import { Config } from '@backstage/config';
import { Entity } from '@backstage/catalog-model';
@@ -10,6 +11,7 @@ import { EntityProvider } from '@backstage/plugin-catalog-backend';
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
import { LocationSpec } from '@backstage/plugin-catalog-backend';
import { Logger } from 'winston';
import { PluginTaskScheduler } from '@backstage/backend-tasks';
import { Response as Response_2 } from 'node-fetch';
import { TaskRunner } from '@backstage/backend-tasks';
@@ -55,8 +57,9 @@ export class BitbucketServerEntityProvider implements EntityProvider {
config: Config,
options: {
logger: Logger;
schedule: TaskRunner;
parser?: BitbucketServerLocationParser;
schedule?: TaskRunner;
scheduler?: PluginTaskScheduler;
},
): BitbucketServerEntityProvider[];
// (undocumented)
@@ -65,6 +68,11 @@ export class BitbucketServerEntityProvider implements EntityProvider {
refresh(logger: Logger): Promise<void>;
}
// @alpha (undocumented)
export const bitbucketServerEntityProviderCatalogModule: (
options?: undefined,
) => BackendFeature;
// @public (undocumented)
export type BitbucketServerListOptions = {
[key: string]: number | undefined;
plugins/catalog-backend-module-bitbucket-server/config.d.ts
Vendored
+10 -3
View File
@@ -14,11 +14,10 @@
* limitations under the License.
*/
import { TaskScheduleDefinitionConfig } from '@backstage/backend-tasks';
export interface Config {
catalog?: {
/**
* List of provider-specific options and attributes
*/
providers?: {
/**
* BitbucketServerEntityProvider configuration
@@ -48,6 +47,10 @@ export interface Config {
*/
projectKey?: RegExp;
};
/**
* (Optional) TaskScheduleDefinition for the refresh.
*/
schedule?: TaskScheduleDefinitionConfig;
}
| Record<
string,
@@ -73,6 +76,10 @@ export interface Config {
*/
projectKey?: RegExp;
};
/**
* (Optional) TaskScheduleDefinition for the refresh.
*/
schedule?: TaskScheduleDefinitionConfig;
}
>;
};
plugins/catalog-backend-module-bitbucket-server/package.json
+10 -5
View File
@@ -6,6 +6,7 @@
"license": "Apache-2.0",
"publishConfig": {
"access": "public",
"alphaTypes": "dist/index.alpha.d.ts",
"main": "dist/index.cjs.js",
"types": "dist/index.d.ts"
},
@@ -23,21 +24,23 @@
],
"scripts": {
"start": "backstage-cli package start",
"build": "backstage-cli package build",
"build": "backstage-cli package build --experimental-type-build",
"lint": "backstage-cli package lint",
"test": "backstage-cli package test",
"clean": "backstage-cli package clean",
"prepack": "backstage-cli package prepack",
"postpack": "backstage-cli package postpack"
"postpack": "backstage-cli package postpack",
"clean": "backstage-cli package clean"
},
"dependencies": {
"@backstage/backend-common": "workspace:^",
"@backstage/backend-plugin-api": "workspace:^",
"@backstage/backend-tasks": "workspace:^",
"@backstage/catalog-model": "workspace:^",
"@backstage/config": "workspace:^",
"@backstage/errors": "workspace:^",
"@backstage/integration": "workspace:^",
"@backstage/plugin-catalog-backend": "workspace:^",
"@backstage/plugin-catalog-node": "workspace:^",
"@types/node-fetch": "^2.5.12",
"node-fetch": "^2.6.7",
"uuid": "^8.0.0",
@@ -46,11 +49,13 @@
"devDependencies": {
"@backstage/backend-test-utils": "workspace:^",
"@backstage/cli": "workspace:^",
"luxon": "^3.0.0",
"msw": "^0.47.0"
},
"files": [
"dist",
"config.d.ts"
"alpha",
"config.d.ts",
"dist"
],
"configSchema": "config.d.ts"
}
plugins/catalog-backend-module-bitbucket-server/src/index.ts
+1
View File
@@ -29,3 +29,4 @@ export type {
} from './lib';
export { BitbucketServerEntityProvider } from './providers';
export type { BitbucketServerLocationParser } from './providers';
export { bitbucketServerEntityProviderCatalogModule } from './service/BitbucketServerEntityProviderCatalogModule';
plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.test.ts
+167 -4
View File
@@ -15,14 +15,18 @@
*/
import { getVoidLogger } from '@backstage/backend-common';
import { TaskInvocationDefinition, TaskRunner } from '@backstage/backend-tasks';
import {
PluginTaskScheduler,
TaskInvocationDefinition,
TaskRunner,
} from '@backstage/backend-tasks';
import { setupRequestMockHandlers } from '@backstage/backend-test-utils';
import { ConfigReader } from '@backstage/config';
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
import { setupRequestMockHandlers } from '@backstage/backend-test-utils';
import { BitbucketServerEntityProvider } from './BitbucketServerEntityProvider';
import { rest } from 'msw';
import { setupServer } from 'msw/node';
import { BitbucketServerPagedResponse } from '../lib/BitbucketServerClient';
import { BitbucketServerEntityProvider } from './BitbucketServerEntityProvider';
import { BitbucketServerPagedResponse } from '../lib';
class PersistingTaskRunner implements TaskRunner {
private tasks: TaskInvocationDefinition[] = [];
@@ -367,4 +371,163 @@ describe('BitbucketServerEntityProvider', () => {
entities: expectedEntities,
});
});
it('fail without schedule and scheduler', () => {
const config = new ConfigReader({
catalog: {
providers: {
bitbucketServer: {
host: 'bitbucket.mycompany.com',
},
},
},
integrations: {
bitbucketServer: [
{
host: 'bitbucket.mycompany.com',
},
],
},
});
expect(() =>
BitbucketServerEntityProvider.fromConfig(config, {
logger,
}),
).toThrow('Either schedule or scheduler must be provided');
});
it('fail with scheduler but no schedule config', () => {
const scheduler = {
createScheduledTaskRunner: (_: any) => jest.fn(),
} as unknown as PluginTaskScheduler;
const config = new ConfigReader({
catalog: {
providers: {
bitbucketServer: {
host: 'bitbucket.mycompany.com',
},
},
},
integrations: {
bitbucketServer: [
{
host: 'bitbucket.mycompany.com',
},
],
},
});
expect(() =>
BitbucketServerEntityProvider.fromConfig(config, {
logger,
scheduler,
}),
).toThrow(
'No schedule provided neither via code nor config for bitbucketServer-provider:default',
);
});
it('apply full update with schedule in config', async () => {
const host = 'bitbucket.mycompany.com';
const config = new ConfigReader({
integrations: {
bitbucketServer: [
{
host: host,
},
],
},
catalog: {
providers: {
bitbucketServer: {
mainProvider: {
host: host,
schedule: {
frequency: 'PT30M',
timeout: {
minutes: 3,
},
},
},
},
},
},
});
const schedule = new PersistingTaskRunner();
const scheduler = {
createScheduledTaskRunner: (_: any) => schedule,
} as unknown as PluginTaskScheduler;
const entityProviderConnection: EntityProviderConnection = {
applyMutation: jest.fn(),
refresh: jest.fn(),
};
const provider = BitbucketServerEntityProvider.fromConfig(config, {
logger,
scheduler,
})[0];
expect(provider.getProviderName()).toEqual(
'bitbucketServer-provider:mainProvider',
);
setupStubs(
[
{ key: 'project-test', repos: ['repo-test'] },
{ key: 'other-project', repos: ['other-repo'] },
],
`https://${host}`,
);
await provider.connect(entityProviderConnection);
const taskDef = schedule.getTasks()[0];
expect(taskDef.id).toEqual('bitbucketServer-provider:mainProvider:refresh');
await (taskDef.fn as () => Promise<void>)();
const expectedEntities = [
{
entity: {
apiVersion: 'backstage.io/v1alpha1',
kind: 'Location',
metadata: {
annotations: {
'backstage.io/managed-by-location': `url:https://${host}/projects/project-test/repos/repo-test/browse/catalog-info.yaml`,
'backstage.io/managed-by-origin-location': `url:https://${host}/projects/project-test/repos/repo-test/browse/catalog-info.yaml`,
},
name: 'generated-77f4323822420990f8c3e3c981d38c2dec4ae3a6',
},
spec: {
presence: 'optional',
target: `https://${host}/projects/project-test/repos/repo-test/browse/catalog-info.yaml`,
type: 'url',
},
},
locationKey: 'bitbucketServer-provider:mainProvider',
},
{
entity: {
apiVersion: 'backstage.io/v1alpha1',
kind: 'Location',
metadata: {
annotations: {
'backstage.io/managed-by-location': `url:https://${host}/projects/other-project/repos/other-repo/browse/catalog-info.yaml`,
'backstage.io/managed-by-origin-location': `url:https://${host}/projects/other-project/repos/other-repo/browse/catalog-info.yaml`,
},
name: 'generated-d8d4944c30c2906dfee172ddda9537f9893b2c0f',
},
spec: {
presence: 'optional',
target: `https://${host}/projects/other-project/repos/other-repo/browse/catalog-info.yaml`,
type: 'url',
},
},
locationKey: 'bitbucketServer-provider:mainProvider',
},
];
expect(entityProviderConnection.applyMutation).toHaveBeenCalledTimes(1);
expect(entityProviderConnection.applyMutation).toHaveBeenCalledWith({
type: 'full',
entities: expectedEntities,
});
});
});
plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts
+34 -18
View File
@@ -14,29 +14,29 @@
* limitations under the License.
*/
import { PluginTaskScheduler, TaskRunner } from '@backstage/backend-tasks';
import { Entity } from '@backstage/catalog-model';
import { Config } from '@backstage/config';
import { InputError } from '@backstage/errors';
import {
BitbucketServerIntegration,
ScmIntegrations,
} from '@backstage/integration';
import {
EntityProvider,
EntityProviderConnection,
} from '@backstage/plugin-catalog-backend';
import { Logger } from 'winston';
import { Config } from '@backstage/config';
import { TaskRunner } from '@backstage/backend-tasks';
import * as uuid from 'uuid';
import {
BitbucketServerLocationParser,
defaultBitbucketServerLocationParser,
} from './BitbucketServerLocationParser';
import {
BitbucketServerIntegration,
ScmIntegrations,
} from '@backstage/integration';
import { BitbucketServerClient, paginated } from '../lib';
import { InputError } from '@backstage/errors';
import {
BitbucketServerEntityProviderConfig,
readProviderConfigs,
} from './BitbucketServerEntityProviderConfig';
import { Entity } from '@backstage/catalog-model';
import {
BitbucketServerLocationParser,
defaultBitbucketServerLocationParser,
} from './BitbucketServerLocationParser';
/**
* Discovers catalog files located in Bitbucket Server.
@@ -58,12 +58,17 @@ export class BitbucketServerEntityProvider implements EntityProvider {
config: Config,
options: {
logger: Logger;
schedule: TaskRunner;
parser?: BitbucketServerLocationParser;
schedule?: TaskRunner;
scheduler?: PluginTaskScheduler;
},
): BitbucketServerEntityProvider[] {
const integrations = ScmIntegrations.fromConfig(config);
if (!options.schedule && !options.scheduler) {
throw new Error('Either schedule or scheduler must be provided.');
}
return readProviderConfigs(config).map(providerConfig => {
const integration = integrations.bitbucketServer.byHost(
providerConfig.host,
@@ -73,11 +78,22 @@ export class BitbucketServerEntityProvider implements EntityProvider {
`No BitbucketServer integration found that matches host ${providerConfig.host}`,
);
}
if (!options.schedule && !providerConfig.schedule) {
throw new Error(
`No schedule provided neither via code nor config for bitbucketServer-provider:${providerConfig.id}.`,
);
}
const taskRunner =
options.schedule ??
options.scheduler!.createScheduledTaskRunner(providerConfig.schedule!);
return new BitbucketServerEntityProvider(
providerConfig,
integration,
options.logger,
options.schedule,
taskRunner,
options.parser,
);
});
@@ -87,7 +103,7 @@ export class BitbucketServerEntityProvider implements EntityProvider {
config: BitbucketServerEntityProviderConfig,
integration: BitbucketServerIntegration,
logger: Logger,
schedule: TaskRunner,
taskRunner: TaskRunner,
parser?: BitbucketServerLocationParser,
) {
this.integration = integration;
@@ -96,13 +112,13 @@ export class BitbucketServerEntityProvider implements EntityProvider {
this.logger = logger.child({
target: this.getProviderName(),
});
this.scheduleFn = this.createScheduleFn(schedule);
this.scheduleFn = this.createScheduleFn(taskRunner);
}
private createScheduleFn(schedule: TaskRunner): () => Promise<void> {
private createScheduleFn(taskRunner: TaskRunner): () => Promise<void> {
return async () => {
const taskId = `${this.getProviderName()}:refresh`;
return schedule.run({
return taskRunner.run({
id: taskId,
fn: async () => {
const logger = this.logger.child({
plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProviderConfig.test.ts
+26 -1
View File
@@ -15,6 +15,7 @@
*/
import { ConfigReader } from '@backstage/config';
import { Duration } from 'luxon';
import { readProviderConfigs } from './BitbucketServerEntityProviderConfig';
describe('readProviderConfigs', () => {
@@ -63,13 +64,22 @@ describe('readProviderConfigs', () => {
host: 'bitbucket2.mycompany.com',
catalogPath: 'custom/path/catalog-info.yaml',
},
thirdProvider: {
host: 'bitbucket3.mycompany.com',
schedule: {
frequency: 'PT30M',
timeout: {
minutes: 3,
},
},
},
},
},
},
});
const providerConfigs = readProviderConfigs(config);
expect(providerConfigs).toHaveLength(2);
expect(providerConfigs).toHaveLength(3);
expect(providerConfigs[0]).toEqual({
id: 'mainProvider',
catalogPath: '/catalog-info.yaml',
@@ -88,6 +98,21 @@ describe('readProviderConfigs', () => {
repoSlug: undefined,
},
});
expect(providerConfigs[2]).toEqual({
id: 'thirdProvider',
catalogPath: '/catalog-info.yaml',
host: 'bitbucket3.mycompany.com',
filters: {
projectKey: undefined,
repoSlug: undefined,
},
schedule: {
frequency: Duration.fromISO('PT30M'),
timeout: {
minutes: 3,
},
},
});
});
it('single provider config with filters', () => {
plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProviderConfig.ts
+10
View File
@@ -14,6 +14,10 @@
* limitations under the License.
*/
import {
readTaskScheduleDefinitionFromConfig,
TaskScheduleDefinition,
} from '@backstage/backend-tasks';
import { Config } from '@backstage/config';
const DEFAULT_CATALOG_PATH = '/catalog-info.yaml';
@@ -27,6 +31,7 @@ export type BitbucketServerEntityProviderConfig = {
projectKey?: RegExp;
repoSlug?: RegExp;
};
schedule?: TaskScheduleDefinition;
};
export function readProviderConfigs(
@@ -60,6 +65,10 @@ function readProviderConfig(
const projectKeyPattern = config.getOptionalString('filters.projectKey');
const repoSlugPattern = config.getOptionalString('filters.repoSlug');
const schedule = config.has('schedule')
? readTaskScheduleDefinitionFromConfig(config.getConfig('schedule'))
: undefined;
return {
id,
host,
@@ -68,5 +77,6 @@ function readProviderConfig(
projectKey: projectKeyPattern ? new RegExp(projectKeyPattern) : undefined,
repoSlug: repoSlugPattern ? new RegExp(repoSlugPattern) : undefined,
},
schedule,
};
}
plugins/catalog-backend-module-bitbucket-server/src/service/BitbucketServerEntityProviderCatalogModule.test.ts
+91
View File
@@ -0,0 +1,91 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { ConfigReader } from '@backstage/config';
import { getVoidLogger } from '@backstage/backend-common';
import {
configServiceRef,
loggerServiceRef,
schedulerServiceRef,
} from '@backstage/backend-plugin-api';
import {
PluginTaskScheduler,
TaskScheduleDefinition,
} from '@backstage/backend-tasks';
import { startTestBackend } from '@backstage/backend-test-utils';
import { catalogProcessingExtensionPoint } from '@backstage/plugin-catalog-node';
import { bitbucketServerEntityProviderCatalogModule } from './BitbucketServerEntityProviderCatalogModule';
import { Duration } from 'luxon';
import { BitbucketServerEntityProvider } from '../providers';
describe('bitbucketServerEntityProviderCatalogModule', () => {
it('should register provider at the catalog extension point', async () => {
let addedProviders: Array<BitbucketServerEntityProvider> | undefined;
let usedSchedule: TaskScheduleDefinition | undefined;
const extensionPoint = {
addEntityProvider: (providers: any) => {
addedProviders = providers;
},
};
const runner = jest.fn();
const scheduler = {
createScheduledTaskRunner: (schedule: TaskScheduleDefinition) => {
usedSchedule = schedule;
return runner;
},
} as unknown as PluginTaskScheduler;
const config = new ConfigReader({
catalog: {
providers: {
bitbucketServer: {
host: 'bitbucket.mycompany.com',
schedule: {
frequency: 'P1M',
timeout: 'PT3M',
},
},
},
},
integrations: {
bitbucketServer: [
{
host: 'bitbucket.mycompany.com',
},
],
},
});
await startTestBackend({
extensionPoints: [[catalogProcessingExtensionPoint, extensionPoint]],
services: [
[configServiceRef, config],
[loggerServiceRef, getVoidLogger()],
[schedulerServiceRef, scheduler],
],
features: [bitbucketServerEntityProviderCatalogModule()],
});
expect(usedSchedule?.frequency).toEqual(Duration.fromISO('P1M'));
expect(usedSchedule?.timeout).toEqual(Duration.fromISO('PT3M'));
expect(addedProviders?.length).toEqual(1);
expect(addedProviders?.pop()?.getProviderName()).toEqual(
'bitbucketServer-provider:default',
);
expect(runner).not.toHaveBeenCalled();
});
});
plugins/catalog-backend-module-bitbucket-server/src/service/BitbucketServerEntityProviderCatalogModule.ts
+52
View File
@@ -0,0 +1,52 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import {
configServiceRef,
createBackendModule,
loggerServiceRef,
loggerToWinstonLogger,
schedulerServiceRef,
} from '@backstage/backend-plugin-api';
import { catalogProcessingExtensionPoint } from '@backstage/plugin-catalog-node';
import { BitbucketServerEntityProvider } from '../providers';
/**
* @alpha
*/
export const bitbucketServerEntityProviderCatalogModule = createBackendModule({
pluginId: 'catalog',
moduleId: 'bitbucketServerEntityProvider',
register(env) {
env.registerInit({
deps: {
catalog: catalogProcessingExtensionPoint,
config: configServiceRef,
logger: loggerServiceRef,
scheduler: schedulerServiceRef,
},
async init({ catalog, config, logger, scheduler }) {
const winstonLogger = loggerToWinstonLogger(logger);
const providers = BitbucketServerEntityProvider.fromConfig(config, {
logger: winstonLogger,
scheduler,
});
catalog.addEntityProvider(providers);
},
});
},
});
plugins/catalog-backend-module-gerrit/api-report.md
+9 -1
View File
@@ -3,10 +3,12 @@
> Do not edit this file. It is a report generated by [API Extractor](https://api-extractor.com/).
```ts
import { BackendFeature } from '@backstage/backend-plugin-api';
import { Config } from '@backstage/config';
import { EntityProvider } from '@backstage/plugin-catalog-backend';
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
import { Logger } from 'winston';
import { PluginTaskScheduler } from '@backstage/backend-tasks';
import { TaskRunner } from '@backstage/backend-tasks';
// @public (undocumented)
@@ -18,7 +20,8 @@ export class GerritEntityProvider implements EntityProvider {
configRoot: Config,
options: {
logger: Logger;
schedule: TaskRunner;
schedule?: TaskRunner;
scheduler?: PluginTaskScheduler;
},
): GerritEntityProvider[];
// (undocumented)
@@ -27,5 +30,10 @@ export class GerritEntityProvider implements EntityProvider {
refresh(logger: Logger): Promise<void>;
}
// @alpha (undocumented)
export const gerritEntityProviderCatalogModule: (
options?: undefined,
) => BackendFeature;
// (No @packageDocumentation comment for this package)
```
plugins/catalog-backend-module-gerrit/package.json
+8 -4
View File
@@ -6,6 +6,7 @@
"license": "Apache-2.0",
"publishConfig": {
"access": "public",
"alphaTypes": "dist/index.alpha.d.ts",
"main": "dist/index.cjs.js",
"types": "dist/index.d.ts"
},
@@ -20,21 +21,23 @@
},
"scripts": {
"start": "backstage-cli package start",
"build": "backstage-cli package build",
"build": "backstage-cli package build --experimental-type-build",
"lint": "backstage-cli package lint",
"test": "backstage-cli package test",
"clean": "backstage-cli package clean",
"prepack": "backstage-cli package prepack",
"postpack": "backstage-cli package postpack"
"postpack": "backstage-cli package postpack",
"clean": "backstage-cli package clean"
},
"dependencies": {
"@backstage/backend-common": "workspace:^",
"@backstage/backend-plugin-api": "workspace:^",
"@backstage/backend-tasks": "workspace:^",
"@backstage/catalog-model": "workspace:^",
"@backstage/config": "workspace:^",
"@backstage/errors": "workspace:^",
"@backstage/integration": "workspace:^",
"@backstage/plugin-catalog-backend": "workspace:^",
"@backstage/plugin-catalog-node": "workspace:^",
"fs-extra": "10.1.0",
"msw": "^0.47.0",
"node-fetch": "^2.6.7",
@@ -44,7 +47,8 @@
"devDependencies": {
"@backstage/backend-test-utils": "workspace:^",
"@backstage/cli": "workspace:^",
"@types/fs-extra": "^9.0.1"
"@types/fs-extra": "^9.0.1",
"luxon": "^3.0.0"
},
"files": [
"dist",
plugins/catalog-backend-module-gerrit/src/index.ts
+1
View File
@@ -15,3 +15,4 @@
*/
export { GerritEntityProvider } from './providers/GerritEntityProvider';
export { gerritEntityProviderCatalogModule } from './service/GerritEntityProviderCatalogModule';
plugins/catalog-backend-module-gerrit/src/providers/GerritEntityProvider.test.ts
+97 -4
View File
@@ -15,14 +15,18 @@
*/
import { getVoidLogger } from '@backstage/backend-common';
import { ConfigReader } from '@backstage/config';
import { TaskInvocationDefinition, TaskRunner } from '@backstage/backend-tasks';
import {
PluginTaskScheduler,
TaskInvocationDefinition,
TaskRunner,
} from '@backstage/backend-tasks';
import { setupRequestMockHandlers } from '@backstage/backend-test-utils';
import { ConfigReader } from '@backstage/config';
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
import { rest } from 'msw';
import fs from 'fs-extra';
import path from 'path';
import { rest } from 'msw';
import { setupServer } from 'msw/node';
import path from 'path';
import { GerritEntityProvider } from './GerritEntityProvider';
const server = setupServer();
@@ -207,4 +211,93 @@ describe('GerritEntityProvider', () => {
}),
).toThrow(/No gerrit integration/);
});
it('fail without schedule and scheduler', () => {
expect(() =>
GerritEntityProvider.fromConfig(config, {
logger,
}),
).toThrow('Either schedule or scheduler must be provided');
});
it('fail with scheduler but no schedule config', () => {
const scheduler = {
createScheduledTaskRunner: (_: any) => jest.fn(),
} as unknown as PluginTaskScheduler;
expect(() =>
GerritEntityProvider.fromConfig(config, {
logger,
scheduler,
}),
).toThrow(
'No schedule provided neither via code nor config for gerrit-provider:active-training',
);
});
it('discovers projects from the api with schedule in config', async () => {
const configWithSchedule = new ConfigReader({
catalog: {
providers: {
gerrit: {
'active-training': {
host: 'g.com',
query: 'state=ACTIVE&prefix=training',
branch: 'main',
schedule: {
frequency: 'PT30M',
timeout: {
minutes: 3,
},
},
},
},
},
},
integrations: {
gerrit: [
{
host: 'g.com',
baseUrl: 'https://g.com/gerrit',
gitilesBaseUrl: 'https:/g.com/gitiles',
},
],
},
});
const scheduler = {
createScheduledTaskRunner: (_: any) => schedule,
} as unknown as PluginTaskScheduler;
const repoBuffer = fs.readFileSync(
path.resolve(__dirname, '__fixtures__/listProjectsBody.txt'),
);
const expected = getJsonFixture('expectedProviderEntities.json');
server.use(
rest.get('https://g.com/gerrit/projects/', (_, res, ctx) =>
res(
ctx.status(200),
ctx.set('Content-Type', 'application/json'),
ctx.body(repoBuffer),
),
),
);
const provider = GerritEntityProvider.fromConfig(configWithSchedule, {
logger,
scheduler,
})[0];
expect(provider.getProviderName()).toEqual(
'gerrit-provider:active-training',
);
await provider.connect(entityProviderConnection);
const taskDef = schedule.getTasks()[0];
expect(taskDef.id).toEqual('gerrit-provider:active-training:refresh');
await (taskDef.fn as () => Promise<void>)();
expect(entityProviderConnection.applyMutation).toHaveBeenCalledWith(
expected,
);
});
});
plugins/catalog-backend-module-gerrit/src/providers/GerritEntityProvider.ts
+23 -7
View File
@@ -14,7 +14,7 @@
* limitations under the License.
*/
import { TaskRunner } from '@backstage/backend-tasks';
import { PluginTaskScheduler, TaskRunner } from '@backstage/backend-tasks';
import { Config } from '@backstage/config';
import { InputError } from '@backstage/errors';
import {
@@ -49,9 +49,14 @@ export class GerritEntityProvider implements EntityProvider {
configRoot: Config,
options: {
logger: Logger;
schedule: TaskRunner;
schedule?: TaskRunner;
scheduler?: PluginTaskScheduler;
},
): GerritEntityProvider[] {
if (!options.schedule && !options.scheduler) {
throw new Error('Either schedule or scheduler must be provided.');
}
const providerConfigs = readGerritConfigs(configRoot);
const integrations = ScmIntegrations.fromConfig(configRoot).gerrit;
const providers: GerritEntityProvider[] = [];
@@ -63,12 +68,23 @@ export class GerritEntityProvider implements EntityProvider {
`No gerrit integration found that matches host ${providerConfig.host}`,
);
}
if (!options.schedule && !providerConfig.schedule) {
throw new Error(
`No schedule provided neither via code nor config for gerrit-provider:${providerConfig.id}.`,
);
}
const taskRunner =
options.schedule ??
options.scheduler!.createScheduledTaskRunner(providerConfig.schedule!);
providers.push(
new GerritEntityProvider(
providerConfig,
integration,
options.logger,
options.schedule,
taskRunner,
),
);
});
@@ -79,14 +95,14 @@ export class GerritEntityProvider implements EntityProvider {
config: GerritProviderConfig,
integration: GerritIntegration,
logger: Logger,
schedule: TaskRunner,
taskRunner: TaskRunner,
) {
this.config = config;
this.integration = integration;
this.logger = logger.child({
target: this.getProviderName(),
});
this.scheduleFn = this.createScheduleFn(schedule);
this.scheduleFn = this.createScheduleFn(taskRunner);
}
getProviderName(): string {
@@ -98,10 +114,10 @@ export class GerritEntityProvider implements EntityProvider {
await this.scheduleFn();
}
private createScheduleFn(schedule: TaskRunner): () => Promise<void> {
private createScheduleFn(taskRunner: TaskRunner): () => Promise<void> {
return async () => {
const taskId = `${this.getProviderName()}:refresh`;
return schedule.run({
return taskRunner.run({
id: taskId,
fn: async () => {
const logger = this.logger.child({
plugins/catalog-backend-module-gerrit/src/providers/config.test.ts
+23 -1
View File
@@ -15,6 +15,7 @@
*/
import { ConfigReader } from '@backstage/config';
import { Duration } from 'luxon';
import { readGerritConfigs } from './config';
describe('readGerritConfigs', () => {
@@ -29,12 +30,24 @@ describe('readGerritConfigs', () => {
query: 'state=ACTIVE',
branch: 'main',
};
const provider3 = {
host: 'gerrit1.com',
query: 'state=ACTIVE',
branch: 'main',
schedule: {
frequency: 'PT30M',
timeout: {
minutes: 3,
},
},
};
const config = {
catalog: {
providers: {
gerrit: {
'active-g1': provider1,
'active-g2': provider2,
'active-g3': provider3,
},
},
},
@@ -42,10 +55,19 @@ describe('readGerritConfigs', () => {
const actual = readGerritConfigs(new ConfigReader(config));
expect(actual).toHaveLength(2);
expect(actual).toHaveLength(3);
expect(actual[0]).toEqual({ ...provider1, id: 'active-g1' });
expect(actual[1]).toEqual({ ...provider2, id: 'active-g2' });
expect(actual[2]).toEqual({
...provider3,
id: 'active-g3',
schedule: {
...provider3.schedule,
frequency: Duration.fromISO(provider3.schedule.frequency),
},
});
});
it('provides default values', () => {
const provider = {
host: 'gerrit1.com',
plugins/catalog-backend-module-gerrit/src/providers/config.ts
+6
View File
@@ -14,6 +14,7 @@
* limitations under the License.
*/
import { readTaskScheduleDefinitionFromConfig } from '@backstage/backend-tasks';
import { Config } from '@backstage/config';
import { GerritProviderConfig } from './types';
@@ -22,11 +23,16 @@ function readGerritConfig(id: string, config: Config): GerritProviderConfig {
const host = config.getString('host');
const query = config.getString('query');
const schedule = config.has('schedule')
? readTaskScheduleDefinitionFromConfig(config.getConfig('schedule'))
: undefined;
return {
branch,
host,
id,
query,
schedule,
};
}
plugins/catalog-backend-module-gerrit/src/providers/types.ts
+3
View File
@@ -14,6 +14,8 @@
* limitations under the License.
*/
import { TaskScheduleDefinition } from '@backstage/backend-tasks';
export type GerritProjectInfo = {
id: string;
name: string;
@@ -28,4 +30,5 @@ export type GerritProviderConfig = {
query: string;
id: string;
branch?: string;
schedule?: TaskScheduleDefinition;
};
plugins/catalog-backend-module-gerrit/src/service/GerritEntityProviderCatalogModule.test.ts
+97
View File
@@ -0,0 +1,97 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { getVoidLogger } from '@backstage/backend-common';
import {
configServiceRef,
loggerServiceRef,
schedulerServiceRef,
} from '@backstage/backend-plugin-api';
import {
PluginTaskScheduler,
TaskScheduleDefinition,
} from '@backstage/backend-tasks';
import { startTestBackend } from '@backstage/backend-test-utils';
import { ConfigReader } from '@backstage/config';
import { catalogProcessingExtensionPoint } from '@backstage/plugin-catalog-node';
import { Duration } from 'luxon';
import { gerritEntityProviderCatalogModule } from './GerritEntityProviderCatalogModule';
import { GerritEntityProvider } from '../providers/GerritEntityProvider';
describe('gerritEntityProviderCatalogModule', () => {
it('should register provider at the catalog extension point', async () => {
let addedProviders: Array<GerritEntityProvider> | undefined;
let usedSchedule: TaskScheduleDefinition | undefined;
const extensionPoint = {
addEntityProvider: (providers: any) => {
addedProviders = providers;
},
};
const runner = jest.fn();
const scheduler = {
createScheduledTaskRunner: (schedule: TaskScheduleDefinition) => {
usedSchedule = schedule;
return runner;
},
} as unknown as PluginTaskScheduler;
const config = new ConfigReader({
catalog: {
providers: {
gerrit: {
test: {
host: 'g.com',
query: 'state=ACTIVE&prefix=training',
branch: 'main',
schedule: {
frequency: 'P1M',
timeout: 'PT3M',
},
},
},
},
},
integrations: {
gerrit: [
{
host: 'g.com',
baseUrl: 'https://g.com/gerrit',
gitilesBaseUrl: 'https:/g.com/gitiles',
},
],
},
});
await startTestBackend({
extensionPoints: [[catalogProcessingExtensionPoint, extensionPoint]],
services: [
[configServiceRef, config],
[loggerServiceRef, getVoidLogger()],
[schedulerServiceRef, scheduler],
],
features: [gerritEntityProviderCatalogModule()],
});
expect(usedSchedule?.frequency).toEqual(Duration.fromISO('P1M'));
expect(usedSchedule?.timeout).toEqual(Duration.fromISO('PT3M'));
expect(addedProviders?.length).toEqual(1);
expect(addedProviders?.pop()?.getProviderName()).toEqual(
'gerrit-provider:test',
);
expect(runner).not.toHaveBeenCalled();
});
});
plugins/catalog-backend-module-gerrit/src/service/GerritEntityProviderCatalogModule.ts
+52
View File
@@ -0,0 +1,52 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import {
configServiceRef,
createBackendModule,
loggerServiceRef,
loggerToWinstonLogger,
schedulerServiceRef,
} from '@backstage/backend-plugin-api';
import { catalogProcessingExtensionPoint } from '@backstage/plugin-catalog-node';
import { GerritEntityProvider } from '../providers/GerritEntityProvider';
/**
* @alpha
*/
export const gerritEntityProviderCatalogModule = createBackendModule({
pluginId: 'catalog',
moduleId: 'gerritEntityProvider',
register(env) {
env.registerInit({
deps: {
catalog: catalogProcessingExtensionPoint,
config: configServiceRef,
logger: loggerServiceRef,
scheduler: schedulerServiceRef,
},
async init({ catalog, config, logger, scheduler }) {
const winstonLogger = loggerToWinstonLogger(logger);
const providers = GerritEntityProvider.fromConfig(config, {
logger: winstonLogger,
scheduler,
});
catalog.addEntityProvider(providers);
},
});
},
});
plugins/catalog-backend-module-gitlab/api-report.md
+9 -1
View File
@@ -3,6 +3,7 @@
> Do not edit this file. It is a report generated by [API Extractor](https://api-extractor.com/).
```ts
import { BackendFeature } from '@backstage/backend-plugin-api';
import { CatalogProcessor } from '@backstage/plugin-catalog-backend';
import { CatalogProcessorEmit } from '@backstage/plugin-catalog-backend';
import { Config } from '@backstage/config';
@@ -10,6 +11,7 @@ import { EntityProvider } from '@backstage/plugin-catalog-backend';
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
import { LocationSpec } from '@backstage/plugin-catalog-backend';
import { Logger } from 'winston';
import { PluginTaskScheduler } from '@backstage/backend-tasks';
import { TaskRunner } from '@backstage/backend-tasks';
// @public
@@ -21,7 +23,8 @@ export class GitlabDiscoveryEntityProvider implements EntityProvider {
config: Config,
options: {
logger: Logger;
schedule: TaskRunner;
schedule?: TaskRunner;
scheduler?: PluginTaskScheduler;
},
): GitlabDiscoveryEntityProvider[];
// (undocumented)
@@ -30,6 +33,11 @@ export class GitlabDiscoveryEntityProvider implements EntityProvider {
refresh(logger: Logger): Promise<void>;
}
// @alpha
export const gitlabDiscoveryEntityProviderCatalogModule: (
options?: undefined,
) => BackendFeature;
// @public
export class GitLabDiscoveryProcessor implements CatalogProcessor {
// (undocumented)
plugins/catalog-backend-module-gitlab/config.d.ts
Vendored
+6 -7
View File
@@ -14,11 +14,10 @@
* limitations under the License.
*/
import { TaskScheduleDefinitionConfig } from '@backstage/backend-tasks';
export interface Config {
catalog?: {
/**
* List of provider-specific options and attributes
*/
providers?: {
/**
* GitlabDiscoveryEntityProvider configuration
@@ -28,27 +27,27 @@ export interface Config {
{
/**
* (Required) Gitlab's host name.
* @visibility backend
*/
host: string;
/**
* (Optional) Gitlab's group[/subgroup] where the discovery is done.
* If not defined the whole project will be scanned.
* @visibility backend
*/
group?: string;
/**
* (Optional) Default branch to read the catalog-info.yaml file.
* If not set, 'master' will be used.
* @visibility backend
*/
branch?: string;
/**
* (Optional) The name used for the catalog file.
* If not set, 'catalog-info.yaml' will be used.
* @visibility backend
*/
entityFilename?: string;
/**
* (Optional) TaskScheduleDefinition for the refresh.
*/
schedule?: TaskScheduleDefinitionConfig;
}
>;
};
plugins/catalog-backend-module-gitlab/package.json
+12 -5
View File
@@ -7,6 +7,7 @@
"license": "Apache-2.0",
"publishConfig": {
"access": "public",
"alphaTypes": "dist/index.alpha.d.ts",
"main": "dist/index.cjs.js",
"types": "dist/index.d.ts"
},
@@ -23,22 +24,24 @@
"backstage"
],
"scripts": {
"build": "backstage-cli package build",
"start": "backstage-cli package start",
"build": "backstage-cli package build --experimental-type-build",
"lint": "backstage-cli package lint",
"test": "backstage-cli package test",
"prepack": "backstage-cli package prepack",
"postpack": "backstage-cli package postpack",
"clean": "backstage-cli package clean",
"start": "backstage-cli package start"
"clean": "backstage-cli package clean"
},
"dependencies": {
"@backstage/backend-common": "workspace:^",
"@backstage/backend-plugin-api": "workspace:^",
"@backstage/backend-tasks": "workspace:^",
"@backstage/catalog-model": "workspace:^",
"@backstage/config": "workspace:^",
"@backstage/errors": "workspace:^",
"@backstage/integration": "workspace:^",
"@backstage/plugin-catalog-backend": "workspace:^",
"@backstage/plugin-catalog-node": "workspace:^",
"@backstage/types": "workspace:^",
"lodash": "^4.17.21",
"msw": "^0.47.0",
@@ -50,9 +53,13 @@
"@backstage/backend-test-utils": "workspace:^",
"@backstage/cli": "workspace:^",
"@types/lodash": "^4.14.151",
"@types/uuid": "^8.0.0"
"@types/uuid": "^8.0.0",
"luxon": "^3.0.0"
},
"files": [
"alpha",
"config.d.ts",
"dist"
]
],
"configSchema": "config.d.ts"
}
plugins/catalog-backend-module-gitlab/src/index.ts
+1
View File
@@ -22,3 +22,4 @@
export { GitLabDiscoveryProcessor } from './GitLabDiscoveryProcessor';
export { GitlabDiscoveryEntityProvider } from './providers';
export { gitlabDiscoveryEntityProviderCatalogModule } from './service/GitlabDiscoveryEntityProviderCatalogModule';
plugins/catalog-backend-module-gitlab/src/lib/types.ts
+3
View File
@@ -14,6 +14,8 @@
* limitations under the License.
*/
import { TaskScheduleDefinition } from '@backstage/backend-tasks';
export type GitlabGroupDescription = {
id: number;
web_url: string;
@@ -36,4 +38,5 @@ export type GitlabProviderConfig = {
branch: string;
catalogFile: string;
projectPattern: RegExp;
schedule?: TaskScheduleDefinition;
};
plugins/catalog-backend-module-gitlab/src/providers/GitlabDiscoveryEntityProvider.test.ts
+114 -3
View File
@@ -15,13 +15,17 @@
*/
import { getVoidLogger } from '@backstage/backend-common';
import { TaskInvocationDefinition, TaskRunner } from '@backstage/backend-tasks';
import {
PluginTaskScheduler,
TaskInvocationDefinition,
TaskRunner,
} from '@backstage/backend-tasks';
import { setupRequestMockHandlers } from '@backstage/backend-test-utils';
import { ConfigReader } from '@backstage/config';
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
import { setupRequestMockHandlers } from '@backstage/backend-test-utils';
import { GitlabDiscoveryEntityProvider } from './GitlabDiscoveryEntityProvider';
import { rest } from 'msw';
import { setupServer } from 'msw/node';
import { GitlabDiscoveryEntityProvider } from './GitlabDiscoveryEntityProvider';
class PersistingTaskRunner implements TaskRunner {
private tasks: TaskInvocationDefinition[] = [];
@@ -338,4 +342,111 @@ describe('GitlabDiscoveryEntityProvider', () => {
],
});
});
it('fail without schedule and scheduler', () => {
const config = new ConfigReader({
integrations: {
gitlab: [
{
host: 'test-gitlab',
apiBaseUrl: 'https://api.gitlab.example/api/v4',
token: '1234',
},
],
},
catalog: {
providers: {
gitlab: {
'test-id': {
host: 'test-gitlab',
group: 'test-group',
},
},
},
},
});
expect(() =>
GitlabDiscoveryEntityProvider.fromConfig(config, {
logger,
}),
).toThrow('Either schedule or scheduler must be provided');
});
it('fail with scheduler but no schedule config', () => {
const scheduler = {
createScheduledTaskRunner: (_: any) => jest.fn(),
} as unknown as PluginTaskScheduler;
const config = new ConfigReader({
integrations: {
gitlab: [
{
host: 'test-gitlab',
apiBaseUrl: 'https://api.gitlab.example/api/v4',
token: '1234',
},
],
},
catalog: {
providers: {
gitlab: {
'test-id': {
host: 'test-gitlab',
group: 'test-group',
},
},
},
},
});
expect(() =>
GitlabDiscoveryEntityProvider.fromConfig(config, {
logger,
scheduler,
}),
).toThrow(
'No schedule provided neither via code nor config for GitlabDiscoveryEntityProvider:test-id',
);
});
it('single simple provider config with schedule in config', async () => {
const schedule = new PersistingTaskRunner();
const scheduler = {
createScheduledTaskRunner: (_: any) => schedule,
} as unknown as PluginTaskScheduler;
const config = new ConfigReader({
integrations: {
gitlab: [
{
host: 'test-gitlab',
apiBaseUrl: 'https://api.gitlab.example/api/v4',
token: '1234',
},
],
},
catalog: {
providers: {
gitlab: {
'test-id': {
host: 'test-gitlab',
group: 'test-group',
schedule: {
frequency: 'PT30M',
timeout: 'PT3M',
},
},
},
},
},
});
const providers = GitlabDiscoveryEntityProvider.fromConfig(config, {
logger,
scheduler,
});
expect(providers).toHaveLength(1);
expect(providers[0].getProviderName()).toEqual(
'GitlabDiscoveryEntityProvider:test-id',
);
});
});
plugins/catalog-backend-module-gitlab/src/providers/GitlabDiscoveryEntityProvider.ts
+29 -9
View File
@@ -14,14 +14,16 @@
* limitations under the License.
*/
import { TaskRunner } from '@backstage/backend-tasks';
import { PluginTaskScheduler, TaskRunner } from '@backstage/backend-tasks';
import { Config } from '@backstage/config';
import { GitLabIntegration, ScmIntegrations } from '@backstage/integration';
import {
EntityProvider,
EntityProviderConnection,
LocationSpec,
locationSpecToLocationEntity,
} from '@backstage/plugin-catalog-backend';
import { LocationSpec } from '@backstage/plugin-catalog-backend';
import * as uuid from 'uuid';
import { Logger } from 'winston';
import {
GitLabClient,
@@ -30,8 +32,6 @@ import {
paginated,
readGitlabConfigs,
} from '../lib';
import * as uuid from 'uuid';
import { locationSpecToLocationEntity } from '@backstage/plugin-catalog-backend';
type Result = {
scanned: number;
@@ -51,8 +51,16 @@ export class GitlabDiscoveryEntityProvider implements EntityProvider {
static fromConfig(
config: Config,
options: { logger: Logger; schedule: TaskRunner },
options: {
logger: Logger;
schedule?: TaskRunner;
scheduler?: PluginTaskScheduler;
},
): GitlabDiscoveryEntityProvider[] {
if (!options.schedule && !options.scheduler) {
throw new Error('Either schedule or scheduler must be provided.');
}
const providerConfigs = readGitlabConfigs(config);
const integrations = ScmIntegrations.fromConfig(config).gitlab;
const providers: GitlabDiscoveryEntityProvider[] = [];
@@ -64,11 +72,23 @@ export class GitlabDiscoveryEntityProvider implements EntityProvider {
`No gitlab integration found that matches host ${providerConfig.host}`,
);
}
if (!options.schedule && !providerConfig.schedule) {
throw new Error(
`No schedule provided neither via code nor config for GitlabDiscoveryEntityProvider:${providerConfig.id}.`,
);
}
const taskRunner =
options.schedule ??
options.scheduler!.createScheduledTaskRunner(providerConfig.schedule!);
providers.push(
new GitlabDiscoveryEntityProvider({
...options,
config: providerConfig,
integration,
taskRunner,
}),
);
});
@@ -79,14 +99,14 @@ export class GitlabDiscoveryEntityProvider implements EntityProvider {
config: GitlabProviderConfig;
integration: GitLabIntegration;
logger: Logger;
schedule: TaskRunner;
taskRunner: TaskRunner;
}) {
this.config = options.config;
this.integration = options.integration;
this.logger = options.logger.child({
target: this.getProviderName(),
});
this.scheduleFn = this.createScheduleFn(options.schedule);
this.scheduleFn = this.createScheduleFn(options.taskRunner);
}
getProviderName(): string {
@@ -98,10 +118,10 @@ export class GitlabDiscoveryEntityProvider implements EntityProvider {
await this.scheduleFn();
}
private createScheduleFn(schedule: TaskRunner): () => Promise<void> {
private createScheduleFn(taskRunner: TaskRunner): () => Promise<void> {
return async () => {
const taskId = `${this.getProviderName()}:refresh`;
return schedule.run({
return taskRunner.run({
id: taskId,
fn: async () => {
const logger = this.logger.child({
plugins/catalog-backend-module-gitlab/src/providers/config.test.ts
+45
View File
@@ -15,6 +15,7 @@
*/
import { ConfigReader } from '@backstage/config';
import { Duration } from 'luxon';
import { readGitlabConfigs } from './config';
describe('config', () => {
@@ -53,6 +54,7 @@ describe('config', () => {
host: 'host',
catalogFile: 'catalog-info.yaml',
projectPattern: /[\s\S]*/,
schedule: undefined,
}),
);
});
@@ -83,6 +85,49 @@ describe('config', () => {
host: 'host',
catalogFile: 'custom-file.yaml',
projectPattern: /[\s\S]*/,
schedule: undefined,
}),
);
});
it('valid config with schedule', () => {
const config = new ConfigReader({
catalog: {
providers: {
gitlab: {
test: {
group: 'group',
host: 'host',
schedule: {
frequency: 'PT30M',
timeout: {
minutes: 3,
},
},
},
},
},
},
});
const result = readGitlabConfigs(config);
expect(result).toHaveLength(1);
result.forEach(r =>
expect(r).toStrictEqual({
id: 'test',
group: 'group',
branch: 'master',
host: 'host',
catalogFile: 'catalog-info.yaml',
projectPattern: /[\s\S]*/,
schedule: {
frequency: Duration.fromISO('PT30M'),
timeout: {
minutes: 3,
},
initialDelay: undefined,
scope: undefined,
},
}),
);
});
plugins/catalog-backend-module-gitlab/src/providers/config.ts
+6
View File
@@ -14,6 +14,7 @@
* limitations under the License.
*/
import { readTaskScheduleDefinitionFromConfig } from '@backstage/backend-tasks';
import { Config } from '@backstage/config';
import { GitlabProviderConfig } from '../lib';
@@ -35,6 +36,10 @@ function readGitlabConfig(id: string, config: Config): GitlabProviderConfig {
config.getOptionalString('projectPattern') ?? /[\s\S]*/,
);
const schedule = config.has('schedule')
? readTaskScheduleDefinitionFromConfig(config.getConfig('schedule'))
: undefined;
return {
id,
group,
@@ -42,6 +47,7 @@ function readGitlabConfig(id: string, config: Config): GitlabProviderConfig {
host,
catalogFile,
projectPattern,
schedule,
};
}
plugins/catalog-backend-module-gitlab/src/service/GitlabDiscoveryEntityProviderCatalogModule.test.ts
+96
View File
@@ -0,0 +1,96 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { getVoidLogger } from '@backstage/backend-common';
import {
configServiceRef,
loggerServiceRef,
schedulerServiceRef,
} from '@backstage/backend-plugin-api';
import {
PluginTaskScheduler,
TaskScheduleDefinition,
} from '@backstage/backend-tasks';
import { startTestBackend } from '@backstage/backend-test-utils';
import { ConfigReader } from '@backstage/config';
import { catalogProcessingExtensionPoint } from '@backstage/plugin-catalog-node';
import { Duration } from 'luxon';
import { gitlabDiscoveryEntityProviderCatalogModule } from './GitlabDiscoveryEntityProviderCatalogModule';
import { GitlabDiscoveryEntityProvider } from '../providers';
describe('gitlabDiscoveryEntityProviderCatalogModule', () => {
it('should register provider at the catalog extension point', async () => {
let addedProviders: Array<GitlabDiscoveryEntityProvider> | undefined;
let usedSchedule: TaskScheduleDefinition | undefined;
const extensionPoint = {
addEntityProvider: (providers: any) => {
addedProviders = providers;
},
};
const runner = jest.fn();
const scheduler = {
createScheduledTaskRunner: (schedule: TaskScheduleDefinition) => {
usedSchedule = schedule;
return runner;
},
} as unknown as PluginTaskScheduler;
const config = new ConfigReader({
integrations: {
gitlab: [
{
host: 'test-gitlab',
apiBaseUrl: 'https://api.gitlab.example/api/v4',
token: '1234',
},
],
},
catalog: {
providers: {
gitlab: {
'test-id': {
host: 'test-gitlab',
group: 'test-group',
schedule: {
frequency: 'P1M',
timeout: 'PT3M',
},
},
},
},
},
});
await startTestBackend({
extensionPoints: [[catalogProcessingExtensionPoint, extensionPoint]],
services: [
[configServiceRef, config],
[loggerServiceRef, getVoidLogger()],
[schedulerServiceRef, scheduler],
],
features: [gitlabDiscoveryEntityProviderCatalogModule()],
});
expect(usedSchedule?.frequency).toEqual(Duration.fromISO('P1M'));
expect(usedSchedule?.timeout).toEqual(Duration.fromISO('PT3M'));
expect(addedProviders?.length).toEqual(1);
expect(addedProviders?.pop()?.getProviderName()).toEqual(
'GitlabDiscoveryEntityProvider:test-id',
);
expect(runner).not.toHaveBeenCalled();
});
});
plugins/catalog-backend-module-gitlab/src/service/GitlabDiscoveryEntityProviderCatalogModule.ts
+53
View File
@@ -0,0 +1,53 @@
/*
* Copyright 2022 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import {
createBackendModule,
loggerToWinstonLogger,
configServiceRef,
loggerServiceRef,
schedulerServiceRef,
} from '@backstage/backend-plugin-api';
import { catalogProcessingExtensionPoint } from '@backstage/plugin-catalog-node';
import { GitlabDiscoveryEntityProvider } from '../providers';
/**
* Registers the GitlabDiscoveryEntityProvider with the catalog processing extension point.
*
* @alpha
*/
export const gitlabDiscoveryEntityProviderCatalogModule = createBackendModule({
pluginId: 'catalog',
moduleId: 'gitlabDiscoveryEntityProvider',
register(env) {
env.registerInit({
deps: {
config: configServiceRef,
catalog: catalogProcessingExtensionPoint,
logger: loggerServiceRef,
scheduler: schedulerServiceRef,
},
async init({ config, catalog, logger, scheduler }) {
catalog.addEntityProvider(
GitlabDiscoveryEntityProvider.fromConfig(config, {
logger: loggerToWinstonLogger(logger),
scheduler,
}),
);
},
});
},
});
yarn.lock
+16
View File
@@ -4289,18 +4289,22 @@ __metadata:
resolution: "@backstage/plugin-catalog-backend-module-aws@workspace:plugins/catalog-backend-module-aws"
dependencies:
"@backstage/backend-common": "workspace:^"
"@backstage/backend-plugin-api": "workspace:^"
"@backstage/backend-tasks": "workspace:^"
"@backstage/backend-test-utils": "workspace:^"
"@backstage/catalog-model": "workspace:^"
"@backstage/cli": "workspace:^"
"@backstage/config": "workspace:^"
"@backstage/errors": "workspace:^"
"@backstage/integration": "workspace:^"
"@backstage/plugin-catalog-backend": "workspace:^"
"@backstage/plugin-catalog-node": "workspace:^"
"@backstage/types": "workspace:^"
"@types/lodash": ^4.14.151
aws-sdk: ^2.840.0
aws-sdk-mock: ^5.2.1
lodash: ^4.17.21
luxon: ^3.0.0
p-limit: ^3.0.2
uuid: ^8.0.0
winston: ^3.2.1
@@ -4313,6 +4317,7 @@ __metadata:
resolution: "@backstage/plugin-catalog-backend-module-azure@workspace:plugins/catalog-backend-module-azure"
dependencies:
"@backstage/backend-common": "workspace:^"
"@backstage/backend-plugin-api": "workspace:^"
"@backstage/backend-tasks": "workspace:^"
"@backstage/backend-test-utils": "workspace:^"
"@backstage/catalog-model": "workspace:^"
@@ -4321,9 +4326,11 @@ __metadata:
"@backstage/errors": "workspace:^"
"@backstage/integration": "workspace:^"
"@backstage/plugin-catalog-backend": "workspace:^"
"@backstage/plugin-catalog-node": "workspace:^"
"@backstage/types": "workspace:^"
"@types/lodash": ^4.14.151
lodash: ^4.17.21
luxon: ^3.0.0
msw: ^0.47.0
node-fetch: ^2.6.7
uuid: ^8.0.0
@@ -4357,6 +4364,7 @@ __metadata:
resolution: "@backstage/plugin-catalog-backend-module-bitbucket-server@workspace:plugins/catalog-backend-module-bitbucket-server"
dependencies:
"@backstage/backend-common": "workspace:^"
"@backstage/backend-plugin-api": "workspace:^"
"@backstage/backend-tasks": "workspace:^"
"@backstage/backend-test-utils": "workspace:^"
"@backstage/catalog-model": "workspace:^"
@@ -4365,7 +4373,9 @@ __metadata:
"@backstage/errors": "workspace:^"
"@backstage/integration": "workspace:^"
"@backstage/plugin-catalog-backend": "workspace:^"
"@backstage/plugin-catalog-node": "workspace:^"
"@types/node-fetch": ^2.5.12
luxon: ^3.0.0
msw: ^0.47.0
node-fetch: ^2.6.7
uuid: ^8.0.0
@@ -4400,6 +4410,7 @@ __metadata:
resolution: "@backstage/plugin-catalog-backend-module-gerrit@workspace:plugins/catalog-backend-module-gerrit"
dependencies:
"@backstage/backend-common": "workspace:^"
"@backstage/backend-plugin-api": "workspace:^"
"@backstage/backend-tasks": "workspace:^"
"@backstage/backend-test-utils": "workspace:^"
"@backstage/catalog-model": "workspace:^"
@@ -4408,8 +4419,10 @@ __metadata:
"@backstage/errors": "workspace:^"
"@backstage/integration": "workspace:^"
"@backstage/plugin-catalog-backend": "workspace:^"
"@backstage/plugin-catalog-node": "workspace:^"
"@types/fs-extra": ^9.0.1
fs-extra: 10.1.0
luxon: ^3.0.0
msw: ^0.47.0
node-fetch: ^2.6.7
uuid: ^8.0.0
@@ -4452,6 +4465,7 @@ __metadata:
resolution: "@backstage/plugin-catalog-backend-module-gitlab@workspace:plugins/catalog-backend-module-gitlab"
dependencies:
"@backstage/backend-common": "workspace:^"
"@backstage/backend-plugin-api": "workspace:^"
"@backstage/backend-tasks": "workspace:^"
"@backstage/backend-test-utils": "workspace:^"
"@backstage/catalog-model": "workspace:^"
@@ -4460,10 +4474,12 @@ __metadata:
"@backstage/errors": "workspace:^"
"@backstage/integration": "workspace:^"
"@backstage/plugin-catalog-backend": "workspace:^"
"@backstage/plugin-catalog-node": "workspace:^"
"@backstage/types": "workspace:^"
"@types/lodash": ^4.14.151
"@types/uuid": ^8.0.0
lodash: ^4.17.21
luxon: ^3.0.0
msw: ^0.47.0
node-fetch: ^2.6.7
uuid: ^8.0.0