Merge branch 'master' into topic/nbs-integration-bitbucket-server

Signed-off-by: Kevin L. <kevin.lecouvey@gmail.com>
2024-07-26 07:30:38 -04:00
parent e4589ef403 2bae52597d
commit d89e8dab86
1092 changed files with 34769 additions and 7421 deletions
@@ -66,8 +66,7 @@ catalog:
 As this provider is not one of the default providers, you will first need to install
 the AWS catalog plugin:

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-aws
 ```

@@ -66,8 +66,7 @@ catalog:
 As this provider is not one of the default providers, you will first need to install
 the AWS catalog plugin:

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-aws
 ```

@@ -102,8 +102,7 @@ It may take some time before the branch is indexed and searchable.
 As this provider is not one of the default providers, you will first need to install
 the Azure catalog plugin:

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-azure
 ```

@@ -102,8 +102,7 @@ It may take some time before the branch is indexed and searchable.
 As this provider is not one of the default providers, you will first need to install
 the Azure catalog plugin:

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-azure
 ```

@@ -18,8 +18,7 @@ Microsoft Graph API.

 The package is not installed by default, therefore you have to add `@backstage/plugin-catalog-backend-module-msgraph` to your backend package.

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-msgraph
 ```

@@ -18,8 +18,7 @@ Microsoft Graph API.

 The package is not installed by default, therefore you have to add `@backstage/plugin-catalog-backend-module-msgraph` to your backend package.

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-msgraph
 ```

@@ -112,6 +111,17 @@ microsoftGraphOrg:
      search: '"description:One" AND ("displayName:Video" OR "displayName:Drive")'
 ```

+If you don't want to only ingest groups matching the `search` and/or `filter` query, but also the groups which are members of the matched groups, you can use the `includeSubGroups` configuration:
+
+```yaml
+microsoftGraphOrg:
+  providerId:
+    group:
+      filter: securityEnabled eq false and mailEnabled eq true and groupTypes/any(c:c+eq+'Unified')
+      search: '"description:One" AND ("displayName:Video" OR "displayName:Drive")'
+      includeSubGroups: true
+```
+
 In addition to these groups, one additional group will be created for your organization.
 All imported groups will be a child of this group.

@@ -181,6 +191,23 @@ microsoftGraphOrg:
      select: ['id', 'displayName', 'description']
 ```

+### Using Provider Config Transformer
+
+Dynamic configuration scaling allows the `msgraph` catalog plugin to adjust its settings at runtime without requiring a redeploy. This feature is useful for scenarios where configuration needs to be updated based on real-time events or changing conditions. For example, you can dynamically adjust synchronization schedules, filters, and search parameters to optimize performance and responsiveness.
+
+:::note
+Adjusting fields that are not used on each scheduled ingestion (e.g., `id`, `schedule`) will have no effect.
+:::
+
+:::warning
+Dynamically changing configuration on the fly can introduce unintended consequences, such as system instability and configuration errors. Please review your transformer carefully to ensure that it is working as anticipated!
+:::
+
+#### Example Use Cases:
+
+- **Filter Scaling**: Adjust filters like `userGroupMember` and `groupFilter` dynamically.
+- **Search Parameter Adjustment**: Change search parameters such as `groupSearch` and `userSelect` on-the-fly.
+
 ### Using Custom Transformers

 Transformers can be configured by extending `microsoftGraphOrgEntityProviderTransformExtensionPoint`. Here is an example:
@@ -192,6 +219,7 @@ import {
  myUserTransformer,
  myGroupTransformer,
  myOrganizationTransformer,
+  myProviderConfigTransformer,
 } from './transformers';

 backend.add(
@@ -213,6 +241,9 @@ backend.add(
          microsoftGraphTransformers.setOrganizationTransformer(
            myOrganizationTransformer,
          );
+          microsoftGraphTransformers.setProviderConfigTransformer(
+            myProviderConfigTransformer,
+          );
          /* highlight-add-end */
        },
      });
@@ -221,7 +252,7 @@ backend.add(
 );
 ```

-The `myUserTransformer`, `myGroupTransformer`, and `myOrganizationTransformer` transformer functions are from the examples in the section below.
+The `myUserTransformer`, `myGroupTransformer`, `myOrganizationTransformer`, and `myProviderConfigTransformer` transformer functions are from the examples in the section below.

 ### Transformer Examples

@@ -233,6 +264,7 @@ import {
  defaultGroupTransformer,
  defaultUserTransformer,
  defaultOrganizationTransformer,
+  MicrosoftGraphProviderConfig,
 } from '@backstage/plugin-catalog-backend-module-msgraph';
 import { GroupEntity, UserEntity } from '@backstage/catalog-model';

@@ -275,6 +307,16 @@ export async function myOrganizationTransformer(
 ): Promise<GroupEntity | undefined> {
  return undefined;
 }
+
+// Example config transformer that expands the group filter to also include 'azure-group-a'
+export async function myProviderConfigTransformer(
+  provider: MicrosoftGraphProviderConfig,
+): Promise<MicrosoftGraphProviderConfig> {
+  if (!provider.groupFilter?.includes('azure-group-a')) {
+    provider.groupFilter = `${provider.groupFilter} or displayName eq 'azure-group-a'`;
+  }
+  return provider;
+}
 ```

 ## Troubleshooting
@@ -19,8 +19,7 @@ backend. The provider is not installed by default, therefore you have to add a
 dependency to `@backstage/plugin-catalog-backend-module-bitbucket-cloud` to your backend
 package.

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-bitbucket-cloud
 ```

@@ -23,7 +23,6 @@ backend. The provider is not installed by default, therefore you have to add a
 dependency to `@backstage/plugin-catalog-backend-module-bitbucket-server` to your backend package.

 ```bash title="From your Backstage root directory"
-# From your Backstage root directory
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-bitbucket-server
 ```

@@ -16,8 +16,7 @@ stored in the root of the matching projects.
 As this provider is not one of the default providers, you will first need to install
 the Gerrit provider plugin:

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-gerrit
 ```

@@ -25,8 +25,7 @@ backend. They are not installed by default, therefore you have to add a
 dependency on `@backstage/plugin-catalog-backend-module-github` to your backend
 package.

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-github
 ```

@@ -273,8 +272,7 @@ backend. They are not installed by default, therefore you have to add a
 dependency on `@backstage/plugin-catalog-backend-module-github` to your backend
 package, plus `@backstage/integration` for the basic credentials management:

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/integration @backstage/plugin-catalog-backend-module-github
 ```

@@ -24,8 +24,7 @@ You will have to add the GitHub Entity provider to your backend as it is not ins
 dependency on `@backstage/plugin-catalog-backend-module-github` to your backend
 package.

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-github
 ```

@@ -40,8 +39,9 @@ backend.add(import('@backstage/plugin-catalog-backend-module-github/alpha'));
 ## Events Support

 The catalog module for GitHub comes with events support enabled.
-This will make it subscribe to its relevant topics (`github.push`)
-and expects these events to be published via the `EventsService`.
+This will make it subscribe to its relevant topics (`github.push`,
+`github.repository`) and expects these events to be published
+via the `EventsService`.

 Additionally, you should install the
 [event router by `events-backend-module-github`](https://github.com/backstage/backstage/tree/master/plugins/events-backend-module-github/README.md)
@@ -55,7 +55,15 @@ You can decide between the following options (extensible):
 - [via HTTP endpoint](https://github.com/backstage/backstage/tree/master/plugins/events-backend/README.md)
 - [via an AWS SQS queue](https://github.com/backstage/backstage/tree/master/plugins/events-backend-module-aws-sqs/README.md)

-You can check the official docs to [configure your webhook](https://docs.github.com/en/developers/webhooks-and-events/webhooks/creating-webhooks) and to [secure your request](https://docs.github.com/en/developers/webhooks-and-events/webhooks/securing-your-webhooks). The webhook will need to be configured to forward `push` events.
+You can check the official docs to [configure your webhook](https://docs.github.com/en/developers/webhooks-and-events/webhooks/creating-webhooks) and to [secure your request](https://docs.github.com/en/developers/webhooks-and-events/webhooks/securing-your-webhooks).
+
+The webhook(s) will need to be configured to react to `push` and
+`repository` events.
+
+Certain actions like `transferred` by the `repository` event type
+will not be supported when you use repository webhooks.
+Please check the GitHubs documentation for these event types and
+its actions.

 ## Configuration

@@ -109,8 +109,8 @@ integrations:
          webhookSecret: ${AUTH_ORG_WEBHOOK_SECRET}
 ```

-:::Note
-Note that in both examples above `apps` is an array which means you can add multiple GitHub Apps using `$include` or environment variables as long as they are each for a different GitHub Org as mentioned under the [Caveats](#caveats) section
+:::note
+Note that in both examples above `apps` is an array which means you can add multiple GitHub Apps using `$include` or environment variables as long as they are each for a different GitHub Org as mentioned under the [Caveats](#caveats) section.
 :::

 ## Limiting the GitHub App installations
@@ -29,8 +29,7 @@ the Processor method (not recommended), it is described separately below.
 The provider is not installed by default, therefore you have to add a dependency
 to `@backstage/plugin-catalog-backend-module-github` to your backend package.

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-github
 ```

@@ -334,8 +333,7 @@ frequency with which they are refreshed, separately from other processors.
 The `GithubOrgReaderProcessor` is not registered by default, so you have to
 install and register it in the catalog plugin:

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-github
 ```

@@ -38,8 +38,7 @@ You will have to add the GitHub Org provider to your backend as it is not instal
 dependency on `@backstage/plugin-catalog-backend-module-github-org` to your backend
 package.

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-github-org
 ```

@@ -20,8 +20,7 @@ This provider can also be configured to ingest GitLab data based on [GitLab Webh
 As this provider is not one of the default providers, you will first need to install
 the gitlab catalog plugin:

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-gitlab
 ```

@@ -25,8 +25,7 @@ This provider can also be configured to ingest GitLab data based on [GitLab Syst

 As this provider is not one of the default providers, you will first need to install the Gitlab provider plugin:

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-gitlab @backstage/plugin-catalog-backend-module-gitlab-org
 ```

@@ -24,8 +24,7 @@ the Processor method (not recommended), it is described separately below.
 The provider is not installed by default, therefore you have to add a dependency
 to `@backstage/plugin-catalog-backend-module-ldap` to your backend package.

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-ldap
 ```

@@ -21,8 +21,7 @@ Backstage in general supports OpenLDAP compatible vendors, as well as Active Dir
 The provider is not installed by default, therefore you have to add a dependency
 to `@backstage/plugin-catalog-backend-module-ldap` to your backend package.

-```bash
-# From your Backstage root directory
+```bash title="From your Backstage root directory"
 yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-ldap
 ```

@@ -86,6 +85,14 @@ catalog:
 These config blocks have a lot of options in them, so we will describe each
 "root" key within the block separately.

+> NOTE:
+>
+> If you want to import users and groups from different LDAP servers, you can define multiple providers with different names.
+> If they should come from the same server, you can define multiple users and groups blocks within the same provider using an array of users / groups.
+> Entries coming from the same block will be able to detect group memberships based on the `memberOf` attribute.
+>
+> If you want only to import users or groups, you can omit the groups or users block.
+
 ### target

 This is the URL of the targeted server, typically on the form