diff --git a/.changeset/msgraph-incremental-initial.md b/.changeset/msgraph-incremental-initial.md index 2a16af91a8..3137c2b57f 100644 --- a/.changeset/msgraph-incremental-initial.md +++ b/.changeset/msgraph-incremental-initial.md @@ -4,4 +4,4 @@ New package: `@backstage/plugin-catalog-backend-module-msgraph-incremental` -Introduces a cursor-based incremental ingestion provider for Microsoft Graph that processes users and groups one page at a time. Unlike `MicrosoftGraphOrgEntityProvider`, this module never holds the full dataset in memory — each burst processes a single page of up to 999 items. The `@odata.nextLink` cursor is persisted so a pod restart resumes from the last completed page rather than starting over. +Introduces a cursor-based incremental ingestion provider for Microsoft Graph that processes users and groups one page at a time. Unlike `MicrosoftGraphOrgEntityProvider`, this module never holds the full dataset in memory — each burst processes a single page (up to 999 users or 100 groups). The `@odata.nextLink` cursor is persisted so a pod restart resumes from the last completed page rather than starting over. diff --git a/plugins/catalog-backend-module-msgraph-incremental/README.md b/plugins/catalog-backend-module-msgraph-incremental/README.md index 60c971e87f..342148f049 100644 --- a/plugins/catalog-backend-module-msgraph-incremental/README.md +++ b/plugins/catalog-backend-module-msgraph-incremental/README.md @@ -9,8 +9,8 @@ AD tenants where holding the full dataset in memory at once is not practical. - **Cursor-based resumption** — the `@odata.nextLink` URL is persisted as the cursor, so a pod restart during ingestion resumes from the last completed page rather than starting over. -- **Memory-efficient** — each burst processes a single page (up to 999 items), - keeping memory usage flat regardless of tenant size. +- **Memory-efficient** — each burst processes a single page (up to 999 users + or 100 groups), keeping memory usage flat regardless of tenant size. - **Photo support** — user profile photos are fetched with a gated pre-check to avoid unnecessary API calls for users without photos. - **Transformer extension point** — user, group, organization, and provider @@ -70,4 +70,5 @@ catalog: | Memory usage | Full dataset in RAM | One page at a time | | Resume on restart | Starts from scratch | Resumes from cursor | | `userGroupMember*` options | Supported | Not supported | +| `groupIncludeSubGroups` | Supported | Not supported | | Suitable for large tenants | No | Yes | diff --git a/plugins/catalog-backend-module-msgraph-incremental/src/MicrosoftGraphIncrementalEntityProvider.ts b/plugins/catalog-backend-module-msgraph-incremental/src/MicrosoftGraphIncrementalEntityProvider.ts index 6c290f17ea..642cf88f44 100644 --- a/plugins/catalog-backend-module-msgraph-incremental/src/MicrosoftGraphIncrementalEntityProvider.ts +++ b/plugins/catalog-backend-module-msgraph-incremental/src/MicrosoftGraphIncrementalEntityProvider.ts @@ -153,8 +153,8 @@ export interface MicrosoftGraphIncrementalEntityProviderOptions { * at a time, and provides them as User and Group entities for the catalog. * * Unlike `MicrosoftGraphOrgEntityProvider`, this provider never holds the full - * dataset in memory at once. Each burst processes a single page of users or - * groups (up to 999 items). This makes it suitable for very large tenants and + * dataset in memory at once. Each burst processes a single page (up to 999 + * users or 100 groups). This makes it suitable for very large tenants and * avoids the memory pressure and long-running task issues of the full-scan * provider. * @@ -166,9 +166,11 @@ export interface MicrosoftGraphIncrementalEntityProviderOptions { * stitching derives `spec.memberOf` on users from these group membership lists. * * @remarks - * `userGroupMemberFilter`, `userGroupMemberSearch`, and `userGroupMemberPath` - * are not supported. Use `userFilter` / `userPath` to restrict which users are - * ingested, and `groupFilter` / `groupSearch` to restrict which groups. + * `userGroupMemberFilter`, `userGroupMemberSearch`, `userGroupMemberPath`, and + * `groupIncludeSubGroups` are not supported. Use `userFilter` / `userPath` to + * restrict which users are ingested, and `groupFilter` / `groupSearch` to + * restrict which groups. Switch to `MicrosoftGraphOrgEntityProvider` if you + * require any of these options. * * @public */