paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 42 Packages Projects Releases Wiki Activity

feat(catalog): add catalog-backend-module-msgraph-incremental plugin

Browse Source 
Introduces a new Backstage backend module that incrementally ingests
users and groups from Microsoft Graph one page at a time, using the
incremental ingestion framework.
Unlike MicrosoftGraphOrgEntityProvider, this module never holds the
full dataset in memory. Each burst processes a single page (up to 999
items), making it suitable for large Azure AD tenants where the
full-scan provider causes memory pressure or OOM failures.
The @odata.nextLink cursor is persisted in the incremental ingestion
marks table, so a pod restart during ingestion resumes from the last
completed page rather than starting over.
Signed-off-by: pillaris <pillaris@adobe.com>
This commit is contained in:
pillaris
2026-04-23 15:56:43 +05:30
parent e2fdc2b61f
commit f1279ea2d6
21 changed files with 2440 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.changeset/msgraph-incremental-initial.md
+7
View File
@@ -0,0 +1,7 @@
---
'@backstage/plugin-catalog-backend-module-msgraph-incremental': patch
---
New package: `@backstage/plugin-catalog-backend-module-msgraph-incremental`
Introduces a cursor-based incremental ingestion provider for Microsoft Graph that processes users and groups one page at a time. Unlike `MicrosoftGraphOrgEntityProvider`, this module never holds the full dataset in memory — each burst processes a single page of up to 999 items. The `@odata.nextLink` cursor is persisted so a pod restart resumes from the last completed page rather than starting over.