paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 31 Packages Projects Releases Wiki Activity

feat: add AwsS3EntityProvider as replacement for AwsS3DiscoveryProcessor

Browse Source 
Add a new provider `AwsS3EntityProvider` as a replacement for the now deprecated
`AwsS3DiscoveryProcessor`.

The new provider will scan configured S3 buckets (with optional) prefix and
add `Location` entities for all discovered catalog files.

These `Location` entities will then be processed as usual.

At each execution, the provider will apply a full mutation, replacing all previous
entities with the new entities/state.

Relates-to: #10183
Signed-off-by: Patrick Jungermann <Patrick.Jungermann@gmail.com>
This commit is contained in:
Patrick Jungermann
2022-03-25 22:47:51 +01:00
parent db21d78179
commit 5969c4b65c
13 changed files with 858 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/integrations/aws-s3/discovery.md
+68 -13
View File
@@ -6,32 +6,55 @@ sidebar_label: Discovery
description: Automatically discovering catalog entities from an AWS S3 Bucket
---
The AWS S3 integration has a special discovery processor for discovering catalog
The AWS S3 integration has a special entity provider for discovering catalog
entities located in an S3 Bucket. If you have a bucket that contains multiple
catalog-info files and want to automatically discover them, you can use this
processor. The processor will crawl your S3 bucket and register entities
catalog files, and you want to automatically discover them, you can use this
provider. The provider will crawl your S3 bucket and register entities
matching the configured path. This can be useful as an alternative to static
locations or manually adding things to the catalog.
To use the discovery processor, you'll need an AWS S3 integration
[set up](locations.md) with an `AWS_ACCESS_KEY`, `AWS_SECRET_ACCESS_KEY`, and
optionally a `roleArn`. Then you can add a location target to the catalog
configuration:
To use the entity provider, you'll need an AWS S3 integration
[set up](locations.md) with `accessKeyId` and `secretAccessKey`, and/or
a `roleArn` or none of these (e.g., profile- or instance-based credentials).
At production deployments, you likely manage these with the permissions attached
to your instance.
At your configuration, you add a provider config per bucket:
```yaml
# app-config.yaml
catalog:
locations:
- type: s3-discovery
target: https://sample-bucket.s3.us-east-2.amazonaws.com/
providers:
awsS3:
yourProviderId: # identifies your dataset / provider independent of config changes
bucketName: sample-bucket
prefix: prefix/ # optional
region: us-east-2 # optional, uses the default region otherwise
```
Note the `s3-discovery` type, as this is not a regular `url` processor.
For simple setups, you can omit the provider ID at the config
which has the same effect as using `default` for it.
As this processor is not one of the default providers, you will first need to install the AWS catalog plugin:
```yaml
# app-config.yaml
catalog:
providers:
awsS3:
# uses "default" as provider ID
bucketName: sample-bucket
prefix: prefix/ # optional
region: us-east-2 # optional, uses the default region otherwise
```
As this provider is not one of the default providers, you will first need to install
the AWS catalog plugin:
```bash
# From the Backstage root directory
yarn install --cwd packages/backend @backstage/plugin-catalog-backend-module-aws
yarn add --cwd packages/backend @backstage/plugin-catalog-backend-module-aws
```
Once you've done that, you'll also need to add the segment below to `packages/backend/src/plugins/catalog.ts`:
@@ -39,6 +62,38 @@ Once you've done that, you'll also need to add the segment below to `packages/ba
```ts
/* packages/backend/src/plugins/catalog.ts */
import { AwsS3EntityProvider } from '@backstage/plugin-catalog-backend-module-aws';
const builder = await CatalogBuilder.create(env);
/** ... other processors and/or providers ... */
builder.addEntityProvider(
...AwsS3EntityProvider.fromConfig(env.config, {
logger: env.logger,
schedule: env.scheduler.createScheduledTaskRunner({
frequency: Duration.fromObject({ minutes: 30 }),
timeout: Duration.fromObject({ minutes: 3 }),
}),
}),
);
```
## Alternative Processor
As alternative to the entity provider `AwsS3EntityProvider`
you can still use the `AwsS3DiscoveryProcessor`.
```yaml
# app-config.yaml
catalog:
locations:
- type: s3-discovery
target: https://sample-bucket.s3.us-east-2.amazonaws.com/prefix/
```
```ts
/* packages/backend/src/plugins/catalog.ts */
import { AwsS3DiscoveryProcessor } from '@backstage/plugin-catalog-backend-module-aws';
const builder = await CatalogBuilder.create(env);