Merge pull request #11604 from goenning/go/add-azuredevops-entity-provider
feat: add `AzureDevOpsEntityProvider` as replacement for `AzureDevOpsDiscoveryProcessor`
This commit is contained in:
Johan Haals
@@ -0,0 +1,65 @@
|
||||
---
|
||||
'@backstage/plugin-catalog-backend-module-azure': patch
|
||||
---
|
||||
|
||||
Add a new provider `AzureDevOpsEntityProvider` as replacement for `AzureDevOpsDiscoveryProcessor`.
|
||||
|
||||
In order to migrate from the `AzureDevOpsDiscoveryProcessor` you need to apply
|
||||
the following changes:
|
||||
|
||||
**Before:**
|
||||
|
||||
```yaml
|
||||
# app-config.yaml
|
||||
|
||||
catalog:
|
||||
locations:
|
||||
- type: azure-discovery
|
||||
target: https://dev.azure.com/myorg/myproject/_git/service-*?path=/catalog-info.yaml
|
||||
```
|
||||
|
||||
```ts
|
||||
/* packages/backend/src/plugins/catalog.ts */
|
||||
|
||||
import { AzureDevOpsDiscoveryProcessor } from '@backstage/plugin-catalog-backend-module-azure';
|
||||
|
||||
const builder = await CatalogBuilder.create(env);
|
||||
/** ... other processors ... */
|
||||
builder.addProcessor(new AzureDevOpsDiscoveryProcessor(env.reader));
|
||||
```
|
||||
|
||||
**After:**
|
||||
|
||||
```yaml
|
||||
# app-config.yaml
|
||||
|
||||
catalog:
|
||||
providers:
|
||||
azureDevOps:
|
||||
anyProviderId:
|
||||
host: selfhostedazure.yourcompany.com # This is only really needed for on-premise user, defaults to dev.azure.com
|
||||
organization: myorg # For on-premise this would be your Collection
|
||||
project: myproject
|
||||
repository: service-*
|
||||
path: /catalog-info.yaml
|
||||
```
|
||||
|
||||
```ts
|
||||
/* packages/backend/src/plugins/catalog.ts */
|
||||
|
||||
import { AzureDevOpsEntityProvider } from '@backstage/plugin-catalog-backend-module-azure';
|
||||
|
||||
const builder = await CatalogBuilder.create(env);
|
||||
/** ... other processors and/or providers ... */
|
||||
builder.addEntityProvider(
|
||||
AzureDevOpsEntityProvider.fromConfig(env.config, {
|
||||
logger: env.logger,
|
||||
schedule: env.scheduler.createScheduledTaskRunner({
|
||||
frequency: { minutes: 30 },
|
||||
timeout: { minutes: 3 },
|
||||
}),
|
||||
}),
|
||||
);
|
||||
```
|
||||
|
||||
Visit [https://backstage.io/docs/integrations/azure/discovery](https://backstage.io/docs/integrations/azure/discovery) for more details and options on configuration.
|
||||
@@ -6,25 +6,95 @@ sidebar_label: Discovery
|
||||
description: Automatically discovering catalog entities from repositories in an Azure DevOps organization
|
||||
---
|
||||
|
||||
The Azure DevOps integration has a special discovery processor for discovering
|
||||
catalog entities within an Azure DevOps. The processor will crawl the Azure
|
||||
The Azure DevOps integration has a special entity provider for discovering
|
||||
catalog entities within an Azure DevOps. The provider will crawl your Azure
|
||||
DevOps organization and register entities matching the configured path. This can
|
||||
be useful as an alternative to static locations or manually adding things to the
|
||||
catalog.
|
||||
|
||||
This guide explains how to install and configure the Azure DevOps Entity Provider (recommended) or the Azure DevOps Processor.
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Code Search Feature
|
||||
|
||||
Azure discovery is driven by the Code Search feature in Azure DevOps, this may not be enabled by default. For Azure
|
||||
DevOps Services you can confirm this by looking at the installed extensions in your Organization Settings. For Azure
|
||||
DevOps Server you'll find this information in your Collection Settings.
|
||||
|
||||
If the Code Search extension is not listed then you can install it from the [Visual Studio Marketplace](https://marketplace.visualstudio.com/items?itemName=ms.vss-code-search&targetId=f9352dac-ba6e-434e-9241-a848a510ce3f&utm_source=vstsproduct&utm_medium=SearchExtStatus).
|
||||
|
||||
### Azure Integration
|
||||
|
||||
Setup [Azure integration](locations.md) with `host` and `token`. Host must be `dev.azure.com` for Cloud users, otherwise set this to your on-premise hostname.
|
||||
|
||||
## Installation
|
||||
|
||||
You will have to add the processors in the catalog initialization code of your
|
||||
backend. They are not installed by default, therefore you have to add a
|
||||
dependency to `@backstage/plugin-catalog-backend-module-azure` to your backend
|
||||
package.
|
||||
At your configuration, you add one or more provider configs:
|
||||
|
||||
```yaml
|
||||
# app-config.yaml
|
||||
catalog:
|
||||
providers:
|
||||
azureDevOps:
|
||||
yourProviderId: # identifies your dataset / provider independent of config changes
|
||||
organization: myorg
|
||||
project: myproject
|
||||
repository: service-* # this will match all repos starting with service-*
|
||||
path: /catalog-info.yaml
|
||||
anotherProviderId: # another identifier
|
||||
organization: myorg
|
||||
project: myproject
|
||||
repository: '*' # this will match all repos starting with service-*
|
||||
path: /src/*/catalog-info.yaml # this will search for files deep inside the /src folder
|
||||
yetAotherProviderId: # guess, what? Another one :)
|
||||
host: selfhostedazure.yourcompany.com
|
||||
organization: myorg
|
||||
project: myproject
|
||||
```
|
||||
|
||||
The parameters available are:
|
||||
|
||||
- `host:` Leave empty for Cloud hosted, otherwise set to your self-hosted instance host.
|
||||
- `organization:` Your Organization slug (or Collection for on-premise users). Required.
|
||||
- `project:` Your project slug. Required.
|
||||
- `repository:` The repository name. Wildcards are supported as show on the examples above. If not set, all repositories will be searched.
|
||||
- `path:` Where to find catalog-info.yaml files. Defaults to /catalog-info.yaml.
|
||||
|
||||
_Note:_ the path parameter follows the same rules as the search on Azure DevOps
|
||||
web interface. For more details visit the
|
||||
[official search documentation](https://docs.microsoft.com/en-us/azure/devops/project/search/get-started-search?view=azure-devops).
|
||||
|
||||
As this provider is not one of the default providers, you will first need to install
|
||||
the Azure catalog plugin:
|
||||
|
||||
```bash
|
||||
# From your Backstage root directory
|
||||
yarn add --cwd packages/backend @backstage/plugin-catalog-backend-module-azure
|
||||
```
|
||||
|
||||
And then add the processors to your catalog builder:
|
||||
Once you've done that, you'll also need to add the segment below to `packages/backend/src/plugins/catalog.ts`:
|
||||
|
||||
```diff
|
||||
/* packages/backend/src/plugins/catalog.ts */
|
||||
+import { AzureDevOpsEntityProvider } from '@backstage/plugin-catalog-backend-module-azure';
|
||||
|
||||
const builder = await CatalogBuilder.create(env);
|
||||
/** ... other processors and/or providers ... */
|
||||
+builder.addEntityProvider(
|
||||
+ AzureDevOpsEntityProvider.fromConfig(env.config, {
|
||||
+ logger: env.logger,
|
||||
+ schedule: env.scheduler.createScheduledTaskRunner({
|
||||
+ frequency: Duration.fromObject({ minutes: 30 }),
|
||||
+ timeout: Duration.fromObject({ minutes: 3 }),
|
||||
+ }),
|
||||
+ }),
|
||||
+);
|
||||
```
|
||||
|
||||
## Alternative Processor
|
||||
|
||||
As an alternative to the entity provider `AzureDevOpsEntityProvider`, you can still use the `AzureDevopsDiscoveryProcessor`.
|
||||
|
||||
```diff
|
||||
// In packages/backend/src/plugins/catalog.ts
|
||||
@@ -37,12 +107,6 @@ And then add the processors to your catalog builder:
|
||||
+ builder.addProcessor(AzureDevOpsDiscoveryProcessor.fromConfig(env.config, { logger: env.logger }));
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
To use the discovery processor, you'll need a Azure integration
|
||||
[set up](locations.md) with a `AZURE_TOKEN`. Then you can add a location target
|
||||
to the catalog configuration:
|
||||
|
||||
```yaml
|
||||
catalog:
|
||||
locations:
|
||||
@@ -70,13 +134,3 @@ When using a custom pattern, the target is composed of five parts:
|
||||
- The path within each repository to find the catalog YAML file. This will
|
||||
usually be `/catalog-info.yaml`, `/src/*/catalog-info.yaml` or a similar
|
||||
variation for catalog files stored in the root directory of each repository.
|
||||
|
||||
_Note:_ the path parameter follows the same rules as the search on Azure DevOps
|
||||
web interface. For more details visit the
|
||||
[official search documentation](https://docs.microsoft.com/en-us/azure/devops/project/search/get-started-search?view=azure-devops).
|
||||
|
||||
Azure discovery is driven by the Code Search feature in Azure DevOps, this may not be enabled by default. For Azure
|
||||
DevOps Services you can confirm this by looking at the installed extensions in your Organization Settings. For Azure
|
||||
DevOps Server you'll find this information in your Collection Settings.
|
||||
|
||||
If the Code Search extension is not listed then you can install it from the [Visual Studio Marketplace](https://marketplace.visualstudio.com/items?itemName=ms.vss-code-search&targetId=f9352dac-ba6e-434e-9241-a848a510ce3f&utm_source=vstsproduct&utm_medium=SearchExtStatus).
|
||||
|
||||
@@ -6,9 +6,12 @@
|
||||
import { CatalogProcessor } from '@backstage/plugin-catalog-backend';
|
||||
import { CatalogProcessorEmit } from '@backstage/plugin-catalog-backend';
|
||||
import { Config } from '@backstage/config';
|
||||
import { EntityProvider } from '@backstage/plugin-catalog-backend';
|
||||
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
|
||||
import { LocationSpec } from '@backstage/plugin-catalog-backend';
|
||||
import { Logger } from 'winston';
|
||||
import { ScmIntegrationRegistry } from '@backstage/integration';
|
||||
import { TaskRunner } from '@backstage/backend-tasks';
|
||||
|
||||
// @public
|
||||
export class AzureDevOpsDiscoveryProcessor implements CatalogProcessor {
|
||||
@@ -32,4 +35,22 @@ export class AzureDevOpsDiscoveryProcessor implements CatalogProcessor {
|
||||
emit: CatalogProcessorEmit,
|
||||
): Promise<boolean>;
|
||||
}
|
||||
|
||||
// @public
|
||||
export class AzureDevOpsEntityProvider implements EntityProvider {
|
||||
// (undocumented)
|
||||
connect(connection: EntityProviderConnection): Promise<void>;
|
||||
// (undocumented)
|
||||
static fromConfig(
|
||||
configRoot: Config,
|
||||
options: {
|
||||
logger: Logger;
|
||||
schedule: TaskRunner;
|
||||
},
|
||||
): AzureDevOpsEntityProvider[];
|
||||
// (undocumented)
|
||||
getProviderName(): string;
|
||||
// (undocumented)
|
||||
refresh(logger: Logger): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
/*
|
||||
* Copyright 2020 The Backstage Authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
interface AzureDevOpsConfig {
|
||||
/**
|
||||
* (Optional) The DevOps host; leave empty for `dev.azure.com`, otherwise set to your self-hosted instance host.
|
||||
* @visibility backend
|
||||
*/
|
||||
host: string;
|
||||
/**
|
||||
* (Required) Your organization slug.
|
||||
* @visibility backend
|
||||
*/
|
||||
organization: string;
|
||||
/**
|
||||
* (Required) Your project slug.
|
||||
* @visibility backend
|
||||
*/
|
||||
project: string;
|
||||
/**
|
||||
* (Optional) The repository name. Wildcards are supported as show on the examples above.
|
||||
* If not set, all repositories will be searched.
|
||||
* @visibility backend
|
||||
*/
|
||||
repository?: string;
|
||||
/**
|
||||
* (Optional) Where to find catalog-info.yaml files. Wildcards are supported.
|
||||
* If not set, defaults to /catalog-info.yaml.
|
||||
* @visibility backend
|
||||
*/
|
||||
path?: string;
|
||||
}
|
||||
|
||||
export interface Config {
|
||||
catalog?: {
|
||||
/**
|
||||
* List of provider-specific options and attributes
|
||||
*/
|
||||
providers?: {
|
||||
/**
|
||||
* AzureDevopsEntityProvider configuration
|
||||
*/
|
||||
azureDevOps?: Record<string, AzureDevOpsConfig>;
|
||||
};
|
||||
};
|
||||
}
|
||||
@@ -35,6 +35,7 @@
|
||||
"dependencies": {
|
||||
"@backstage/backend-common": "^0.13.6-next.1",
|
||||
"@backstage/catalog-model": "^1.0.3-next.0",
|
||||
"@backstage/backend-tasks": "^0.3.2-next.0",
|
||||
"@backstage/config": "^1.0.1",
|
||||
"@backstage/errors": "^1.0.0",
|
||||
"@backstage/integration": "^1.2.1-next.1",
|
||||
@@ -42,6 +43,7 @@
|
||||
"@backstage/types": "^1.0.0",
|
||||
"lodash": "^4.17.21",
|
||||
"msw": "^0.42.0",
|
||||
"uuid": "^8.0.0",
|
||||
"node-fetch": "^2.6.7",
|
||||
"winston": "^3.2.1"
|
||||
},
|
||||
@@ -51,6 +53,8 @@
|
||||
"@types/lodash": "^4.14.151"
|
||||
},
|
||||
"files": [
|
||||
"dist"
|
||||
]
|
||||
"dist",
|
||||
"config.d.ts"
|
||||
],
|
||||
"configSchema": "config.d.ts"
|
||||
}
|
||||
|
||||
@@ -20,4 +20,5 @@
|
||||
* @packageDocumentation
|
||||
*/
|
||||
|
||||
export { AzureDevOpsDiscoveryProcessor } from './AzureDevOpsDiscoveryProcessor';
|
||||
export { AzureDevOpsDiscoveryProcessor } from './processors';
|
||||
export { AzureDevOpsEntityProvider } from './providers';
|
||||
|
||||
+2
-2
@@ -21,9 +21,9 @@ import {
|
||||
AzureDevOpsDiscoveryProcessor,
|
||||
parseUrl,
|
||||
} from './AzureDevOpsDiscoveryProcessor';
|
||||
import { codeSearch } from './lib';
|
||||
import { codeSearch } from '../lib';
|
||||
|
||||
jest.mock('./lib');
|
||||
jest.mock('../lib');
|
||||
const mockCodeSearch = codeSearch as jest.MockedFunction<typeof codeSearch>;
|
||||
|
||||
describe('AzureDevOpsDiscoveryProcessor', () => {
|
||||
+1
-1
@@ -26,7 +26,7 @@ import {
|
||||
processingResult,
|
||||
} from '@backstage/plugin-catalog-backend';
|
||||
import { Logger } from 'winston';
|
||||
import { codeSearch } from './lib';
|
||||
import { codeSearch } from '../lib';
|
||||
|
||||
/**
|
||||
* Extracts repositories out of an Azure DevOps org.
|
||||
@@ -0,0 +1,17 @@
|
||||
/*
|
||||
* Copyright 2022 The Backstage Authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
export { AzureDevOpsDiscoveryProcessor } from './AzureDevOpsDiscoveryProcessor';
|
||||
@@ -0,0 +1,195 @@
|
||||
/*
|
||||
* Copyright 2022 The Backstage Authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
import { getVoidLogger } from '@backstage/backend-common';
|
||||
import { TaskInvocationDefinition, TaskRunner } from '@backstage/backend-tasks';
|
||||
import { ConfigReader } from '@backstage/config';
|
||||
import { EntityProviderConnection } from '@backstage/plugin-catalog-backend';
|
||||
import { CodeSearchResultItem } from '../lib';
|
||||
import { AzureDevOpsEntityProvider } from './AzureDevOpsEntityProvider';
|
||||
import { codeSearch } from '../lib';
|
||||
|
||||
jest.mock('../lib');
|
||||
const mockCodeSearch = codeSearch as jest.MockedFunction<typeof codeSearch>;
|
||||
|
||||
class PersistingTaskRunner implements TaskRunner {
|
||||
private tasks: TaskInvocationDefinition[] = [];
|
||||
|
||||
getTasks() {
|
||||
return this.tasks;
|
||||
}
|
||||
|
||||
run(task: TaskInvocationDefinition): Promise<void> {
|
||||
this.tasks.push(task);
|
||||
return Promise.resolve(undefined);
|
||||
}
|
||||
}
|
||||
|
||||
const logger = getVoidLogger();
|
||||
|
||||
describe('AzureDevOpsEntityProvider', () => {
|
||||
afterEach(() => {
|
||||
mockCodeSearch.mockClear();
|
||||
});
|
||||
|
||||
const expectMutation = async (
|
||||
providerId: string,
|
||||
providerConfig: object,
|
||||
codeSearchResults: CodeSearchResultItem[],
|
||||
expectedBaseUrl: string,
|
||||
names: Record<string, string>,
|
||||
integrationConfig?: object,
|
||||
) => {
|
||||
const config = new ConfigReader({
|
||||
integrations: {
|
||||
azure: integrationConfig ? [integrationConfig] : [],
|
||||
},
|
||||
catalog: {
|
||||
providers: {
|
||||
azureDevOps: {
|
||||
[providerId]: providerConfig,
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
mockCodeSearch.mockResolvedValueOnce(codeSearchResults);
|
||||
|
||||
const schedule = new PersistingTaskRunner();
|
||||
const entityProviderConnection: EntityProviderConnection = {
|
||||
applyMutation: jest.fn(),
|
||||
};
|
||||
|
||||
const provider = AzureDevOpsEntityProvider.fromConfig(config, {
|
||||
logger,
|
||||
schedule,
|
||||
})[0];
|
||||
expect(provider.getProviderName()).toEqual(
|
||||
`AzureDevOpsEntityProvider:${providerId}`,
|
||||
);
|
||||
|
||||
await provider.connect(entityProviderConnection);
|
||||
|
||||
const taskDef = schedule.getTasks()[0];
|
||||
expect(taskDef.id).toEqual(
|
||||
`AzureDevOpsEntityProvider:${providerId}:refresh`,
|
||||
);
|
||||
await (taskDef.fn as () => Promise<void>)();
|
||||
|
||||
const expectedEntities = codeSearchResults.map(item => {
|
||||
const url = encodeURI(
|
||||
`${expectedBaseUrl}/_git/${item.repository.name}?path=${item.path}`,
|
||||
);
|
||||
return {
|
||||
entity: {
|
||||
apiVersion: 'backstage.io/v1alpha1',
|
||||
kind: 'Location',
|
||||
metadata: {
|
||||
annotations: {
|
||||
'backstage.io/managed-by-location': `url:${url}`,
|
||||
'backstage.io/managed-by-origin-location': `url:${url}`,
|
||||
},
|
||||
name: names[`${item.repository.name}?path=${item.path}`],
|
||||
},
|
||||
spec: {
|
||||
presence: 'required',
|
||||
target: `${url}`,
|
||||
type: 'url',
|
||||
},
|
||||
},
|
||||
locationKey: `AzureDevOpsEntityProvider:${providerId}`,
|
||||
};
|
||||
});
|
||||
|
||||
expect(entityProviderConnection.applyMutation).toBeCalledWith({
|
||||
type: 'full',
|
||||
entities: expectedEntities,
|
||||
});
|
||||
};
|
||||
|
||||
// eslint-disable-next-line jest/expect-expect
|
||||
it('no mutation when repos are empty', async () => {
|
||||
return expectMutation(
|
||||
'allRepos',
|
||||
{
|
||||
organization: 'myorganization',
|
||||
project: 'myproject',
|
||||
},
|
||||
[],
|
||||
'https://dev.azure.com/myorganization/myproject',
|
||||
{},
|
||||
);
|
||||
});
|
||||
|
||||
// eslint-disable-next-line jest/expect-expect
|
||||
it('single mutation when repos have 1 file found', async () => {
|
||||
return expectMutation(
|
||||
'allReposSingleFile',
|
||||
{
|
||||
organization: 'myorganization',
|
||||
project: 'myproject',
|
||||
},
|
||||
[
|
||||
{
|
||||
fileName: 'catalog-info.yaml',
|
||||
path: '/catalog-info.yaml',
|
||||
repository: {
|
||||
name: 'myrepo',
|
||||
},
|
||||
},
|
||||
],
|
||||
'https://dev.azure.com/myorganization/myproject',
|
||||
{
|
||||
'myrepo?path=/catalog-info.yaml':
|
||||
'generated-87865246726bb12a8c4fb4f914443f1fbb91648c',
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
// eslint-disable-next-line jest/expect-expect
|
||||
it('single mutation when multiple repos have multiple files', async () => {
|
||||
return expectMutation(
|
||||
'allReposMultipleFiles',
|
||||
{
|
||||
organization: 'myorganization',
|
||||
project: 'myproject',
|
||||
},
|
||||
[
|
||||
{
|
||||
fileName: 'catalog-info.yaml',
|
||||
path: '/catalog-info.yaml',
|
||||
repository: {
|
||||
name: 'myrepo',
|
||||
},
|
||||
},
|
||||
{
|
||||
fileName: 'catalog-info.yaml',
|
||||
path: '/catalog-info.yaml',
|
||||
repository: {
|
||||
name: 'myotherrepo',
|
||||
},
|
||||
},
|
||||
],
|
||||
'https://dev.azure.com/myorganization/myproject',
|
||||
{
|
||||
'myrepo?path=/catalog-info.yaml':
|
||||
'generated-87865246726bb12a8c4fb4f914443f1fbb91648c',
|
||||
'myotherrepo?path=/catalog-info.yaml':
|
||||
'generated-2deccac384c34d0dca37be0ebb4b1c8cf6913fe1',
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,167 @@
|
||||
/*
|
||||
* Copyright 2022 The Backstage Authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
import { TaskRunner } from '@backstage/backend-tasks';
|
||||
import { Config } from '@backstage/config';
|
||||
import { AzureIntegration, ScmIntegrations } from '@backstage/integration';
|
||||
import {
|
||||
EntityProvider,
|
||||
EntityProviderConnection,
|
||||
LocationSpec,
|
||||
locationSpecToLocationEntity,
|
||||
} from '@backstage/plugin-catalog-backend';
|
||||
import { readAzureDevOpsConfigs } from './config';
|
||||
import { Logger } from 'winston';
|
||||
import { AzureDevOpsConfig } from './types';
|
||||
import * as uuid from 'uuid';
|
||||
import { codeSearch, CodeSearchResultItem } from '../lib';
|
||||
|
||||
/**
|
||||
* Provider which discovers catalog files within an Azure DevOps repositories.
|
||||
*
|
||||
* Use `AzureDevOpsEntityProvider.fromConfig(...)` to create instances.
|
||||
*
|
||||
* @public
|
||||
*/
|
||||
export class AzureDevOpsEntityProvider implements EntityProvider {
|
||||
private readonly logger: Logger;
|
||||
private readonly scheduleFn: () => Promise<void>;
|
||||
private connection?: EntityProviderConnection;
|
||||
|
||||
static fromConfig(
|
||||
configRoot: Config,
|
||||
options: {
|
||||
logger: Logger;
|
||||
schedule: TaskRunner;
|
||||
},
|
||||
): AzureDevOpsEntityProvider[] {
|
||||
const providerConfigs = readAzureDevOpsConfigs(configRoot);
|
||||
|
||||
return providerConfigs.map(providerConfig => {
|
||||
const integration = ScmIntegrations.fromConfig(configRoot).azure.byHost(
|
||||
providerConfig.host,
|
||||
);
|
||||
|
||||
if (!integration) {
|
||||
throw new Error(
|
||||
`There is no Azure integration for host ${providerConfig.host}. Please add a configuration entry for it under integrations.azure`,
|
||||
);
|
||||
}
|
||||
|
||||
return new AzureDevOpsEntityProvider(
|
||||
providerConfig,
|
||||
integration,
|
||||
options.logger,
|
||||
options.schedule,
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
private constructor(
|
||||
private readonly config: AzureDevOpsConfig,
|
||||
private readonly integration: AzureIntegration,
|
||||
logger: Logger,
|
||||
schedule: TaskRunner,
|
||||
) {
|
||||
this.logger = logger.child({
|
||||
target: this.getProviderName(),
|
||||
});
|
||||
|
||||
this.scheduleFn = this.createScheduleFn(schedule);
|
||||
}
|
||||
|
||||
private createScheduleFn(schedule: TaskRunner): () => Promise<void> {
|
||||
return async () => {
|
||||
const taskId = `${this.getProviderName()}:refresh`;
|
||||
return schedule.run({
|
||||
id: taskId,
|
||||
fn: async () => {
|
||||
const logger = this.logger.child({
|
||||
class: AzureDevOpsEntityProvider.prototype.constructor.name,
|
||||
taskId,
|
||||
taskInstanceId: uuid.v4(),
|
||||
});
|
||||
|
||||
try {
|
||||
await this.refresh(logger);
|
||||
} catch (error) {
|
||||
logger.error(error);
|
||||
}
|
||||
},
|
||||
});
|
||||
};
|
||||
}
|
||||
|
||||
/** {@inheritdoc @backstage/plugin-catalog-backend#EntityProvider.getProviderName} */
|
||||
getProviderName(): string {
|
||||
return `AzureDevOpsEntityProvider:${this.config.id}`;
|
||||
}
|
||||
|
||||
/** {@inheritdoc @backstage/plugin-catalog-backend#EntityProvider.connect} */
|
||||
async connect(connection: EntityProviderConnection): Promise<void> {
|
||||
this.connection = connection;
|
||||
await this.scheduleFn();
|
||||
}
|
||||
|
||||
async refresh(logger: Logger) {
|
||||
if (!this.connection) {
|
||||
throw new Error('Not initialized');
|
||||
}
|
||||
|
||||
logger.info('Discovering Azure DevOps catalog files');
|
||||
|
||||
const files = await codeSearch(
|
||||
this.integration.config,
|
||||
this.config.organization,
|
||||
this.config.project,
|
||||
this.config.repository,
|
||||
this.config.path,
|
||||
);
|
||||
|
||||
logger.info(`Discovered ${files.length} catalog files`);
|
||||
|
||||
const locations = files.map(key => this.createLocationSpec(key));
|
||||
|
||||
await this.connection.applyMutation({
|
||||
type: 'full',
|
||||
entities: locations.map(location => {
|
||||
return {
|
||||
locationKey: this.getProviderName(),
|
||||
entity: locationSpecToLocationEntity({ location }),
|
||||
};
|
||||
}),
|
||||
});
|
||||
|
||||
logger.info(
|
||||
`Committed ${locations.length} locations for AzureDevOps catalog files`,
|
||||
);
|
||||
}
|
||||
|
||||
private createLocationSpec(file: CodeSearchResultItem): LocationSpec {
|
||||
return {
|
||||
type: 'url',
|
||||
target: this.createObjectUrl(file),
|
||||
presence: 'required',
|
||||
};
|
||||
}
|
||||
|
||||
private createObjectUrl(file: CodeSearchResultItem): string {
|
||||
const baseUrl = `https://${this.config.host}/${this.config.organization}/${this.config.project}`;
|
||||
return encodeURI(
|
||||
`${baseUrl}/_git/${file.repository.name}?path=${file.path}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
/*
|
||||
* Copyright 2020 The Backstage Authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
import { ConfigReader } from '@backstage/config';
|
||||
import { readAzureDevOpsConfigs } from './config';
|
||||
|
||||
describe('readAzureDevOpsConfigs', () => {
|
||||
it('reads all provider configs and set default values', () => {
|
||||
const provider1 = {
|
||||
host: 'azure.mycompany.com',
|
||||
organization: 'mycompany',
|
||||
project: 'myproject',
|
||||
};
|
||||
const provider2 = {
|
||||
organization: 'mycompany',
|
||||
project: 'myproject',
|
||||
};
|
||||
const provider3 = {
|
||||
organization: 'mycompany',
|
||||
project: 'myproject',
|
||||
repository: 'service-*',
|
||||
};
|
||||
const config = {
|
||||
catalog: {
|
||||
providers: {
|
||||
azureDevOps: { provider1, provider2, provider3 },
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
const actual = readAzureDevOpsConfigs(new ConfigReader(config));
|
||||
|
||||
expect(actual).toHaveLength(3);
|
||||
expect(actual[0]).toEqual({
|
||||
...provider1,
|
||||
path: '/catalog-info.yaml',
|
||||
repository: '*',
|
||||
id: 'provider1',
|
||||
});
|
||||
expect(actual[1]).toEqual({
|
||||
...provider2,
|
||||
host: 'dev.azure.com',
|
||||
path: '/catalog-info.yaml',
|
||||
repository: '*',
|
||||
id: 'provider2',
|
||||
});
|
||||
expect(actual[2]).toEqual({
|
||||
...provider3,
|
||||
host: 'dev.azure.com',
|
||||
path: '/catalog-info.yaml',
|
||||
id: 'provider3',
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,53 @@
|
||||
/*
|
||||
* Copyright 2022 The Backstage Authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
import { Config } from '@backstage/config';
|
||||
import { AzureDevOpsConfig } from './types';
|
||||
|
||||
export function readAzureDevOpsConfigs(config: Config): AzureDevOpsConfig[] {
|
||||
const configs: AzureDevOpsConfig[] = [];
|
||||
|
||||
const providerConfigs = config.getOptionalConfig(
|
||||
'catalog.providers.azureDevOps',
|
||||
);
|
||||
|
||||
if (!providerConfigs) {
|
||||
return configs;
|
||||
}
|
||||
|
||||
for (const id of providerConfigs.keys()) {
|
||||
configs.push(readAzureDevOpsConfig(id, providerConfigs.getConfig(id)));
|
||||
}
|
||||
|
||||
return configs;
|
||||
}
|
||||
|
||||
function readAzureDevOpsConfig(id: string, config: Config): AzureDevOpsConfig {
|
||||
const organization = config.getString('organization');
|
||||
const project = config.getString('project');
|
||||
const host = config.getOptionalString('host') || 'dev.azure.com';
|
||||
const repository = config.getOptionalString('repository') || '*';
|
||||
const path = config.getOptionalString('path') || '/catalog-info.yaml';
|
||||
|
||||
return {
|
||||
id,
|
||||
host,
|
||||
organization,
|
||||
project,
|
||||
repository,
|
||||
path,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
/*
|
||||
* Copyright 2022 The Backstage Authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
export { AzureDevOpsEntityProvider } from './AzureDevOpsEntityProvider';
|
||||
@@ -0,0 +1,24 @@
|
||||
/*
|
||||
* Copyright 2022 The Backstage Authors
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
export type AzureDevOpsConfig = {
|
||||
id: string;
|
||||
host: string;
|
||||
organization: string;
|
||||
project: string;
|
||||
repository: string;
|
||||
path: string;
|
||||
};
|
||||
Reference in New Issue
Block a user