Merge pull request #10480 from Bonial-International-GmbH/PJ_awsS3-provider

feat: add `AwsS3EntityProvider` as replacement for `AwsS3DiscoveryProcessor`

docs/integrations/aws-s3/discovery.md

@@ -6,32 +6,55 @@ sidebar_label: Discovery
 description: Automatically discovering catalog entities from an AWS S3 Bucket
 ---
 
-The AWS S3 integration has a special discovery processor for discovering catalog
+The AWS S3 integration has a special entity provider for discovering catalog
 entities located in an S3 Bucket. If you have a bucket that contains multiple
-catalog-info files and want to automatically discover them, you can use this
-processor. The processor will crawl your S3 bucket and register entities
+catalog files, and you want to automatically discover them, you can use this
+provider. The provider will crawl your S3 bucket and register entities
 matching the configured path. This can be useful as an alternative to static
 locations or manually adding things to the catalog.
 
-To use the discovery processor, you'll need an AWS S3 integration
-[set up](locations.md) with an `AWS_ACCESS_KEY`, `AWS_SECRET_ACCESS_KEY`, and
-optionally a `roleArn`. Then you can add a location target to the catalog
-configuration:
+To use the entity provider, you'll need an AWS S3 integration
+[set up](locations.md) with `accessKeyId` and `secretAccessKey`, and/or
+a `roleArn` or none of these (e.g., profile- or instance-based credentials).
+
+At production deployments, you likely manage these with the permissions attached
+to your instance.
+
+At your configuration, you add a provider config per bucket:
 
 ```yaml
+# app-config.yaml
+
 catalog:
-  locations:
-    - type: s3-discovery
-      target: https://sample-bucket.s3.us-east-2.amazonaws.com/
+  providers:
+    awsS3:
+      yourProviderId: # identifies your dataset / provider independent of config changes
+        bucketName: sample-bucket
+        prefix: prefix/ # optional
+        region: us-east-2 # optional, uses the default region otherwise
 ```
 
-Note the `s3-discovery` type, as this is not a regular `url` processor.
+For simple setups, you can omit the provider ID at the config
+which has the same effect as using `default` for it.
 
-As this processor is not one of the default providers, you will first need to install the AWS catalog plugin:
+```yaml
+# app-config.yaml
+
+catalog:
+  providers:
+    awsS3:
+      # uses "default" as provider ID
+      bucketName: sample-bucket
+      prefix: prefix/ # optional
+      region: us-east-2 # optional, uses the default region otherwise
+```
+
+As this provider is not one of the default providers, you will first need to install
+the AWS catalog plugin:
 
 ```bash
 # From the Backstage root directory
-yarn install --cwd packages/backend @backstage/plugin-catalog-backend-module-aws
+yarn add --cwd packages/backend @backstage/plugin-catalog-backend-module-aws
 ```
 
 Once you've done that, you'll also need to add the segment below to `packages/backend/src/plugins/catalog.ts`:
@@ -39,6 +62,38 @@ Once you've done that, you'll also need to add the segment below to `packages/ba
 ```ts
 /* packages/backend/src/plugins/catalog.ts */
 
+import { AwsS3EntityProvider } from '@backstage/plugin-catalog-backend-module-aws';
+
+const builder = await CatalogBuilder.create(env);
+/** ... other processors and/or providers ... */
+builder.addEntityProvider(
+  ...AwsS3EntityProvider.fromConfig(env.config, {
+    logger: env.logger,
+    schedule: env.scheduler.createScheduledTaskRunner({
+      frequency: Duration.fromObject({ minutes: 30 }),
+      timeout: Duration.fromObject({ minutes: 3 }),
+    }),
+  }),
+);
+```
+
+## Alternative Processor
+
+As alternative to the entity provider `AwsS3EntityProvider`
+you can still use the `AwsS3DiscoveryProcessor`.
+
+```yaml
+# app-config.yaml
+
+catalog:
+  locations:
+    - type: s3-discovery
+      target: https://sample-bucket.s3.us-east-2.amazonaws.com/prefix/
+```
+
+```ts
+/* packages/backend/src/plugins/catalog.ts */
+
 import { AwsS3DiscoveryProcessor } from '@backstage/plugin-catalog-backend-module-aws';
 
 const builder = await CatalogBuilder.create(env);