From 637a3de8d87292cf999e221ff2008c48cdecdae9 Mon Sep 17 00:00:00 2001 From: abhishekbvs Date: Mon, 20 Oct 2025 01:58:25 +0530 Subject: [PATCH 1/5] feat: add configurable GitHub API page sizes - Add pageSizes configuration for GitHub providers - Document pageSizes configuration Related to #31437 Signed-off-by: abhishekbvs --- .changeset/github-api-page-sizes.md | 83 +++++++++++ docs/integrations/github/discovery.md | 20 +++ docs/integrations/github/org.md | 31 ++++ .../src/module.test.ts | 102 +++++++++++++ .../src/module.ts | 17 +++ .../catalog-backend-module-github/config.d.ts | 78 ++++++++++ .../report.api.md | 13 ++ .../src/index.ts | 2 + .../src/lib/github.test.ts | 140 ++++++++++++++++++ .../src/lib/github.ts | 137 +++++++++++++---- .../src/lib/index.ts | 2 + .../src/providers/GithubEntityProvider.ts | 14 +- .../GithubEntityProviderConfig.test.ts | 64 ++++++++ .../providers/GithubEntityProviderConfig.ts | 10 ++ .../providers/GithubMultiOrgEntityProvider.ts | 35 +++++ 15 files changed, 719 insertions(+), 29 deletions(-) create mode 100644 .changeset/github-api-page-sizes.md diff --git a/.changeset/github-api-page-sizes.md b/.changeset/github-api-page-sizes.md new file mode 100644 index 0000000000..724adbd764 --- /dev/null +++ b/.changeset/github-api-page-sizes.md @@ -0,0 +1,83 @@ +--- +'@backstage/plugin-catalog-backend-module-github': minor +'@backstage/plugin-catalog-backend-module-github-org': minor +--- + +Added configurable page sizes for GitHub GraphQL API queries to prevent `RESOURCE_LIMITS_EXCEEDED` errors with large GitHub organizations. + +**Default Values Changed:** + +To prevent `RESOURCE_LIMITS_EXCEEDED` errors by default, the page sizes have been reduced to 50% of previous values: + +- `teams`: 50 → **25** +- `teamMembers`: 100 → **50** +- `organizationMembers`: 100 → **50** +- `repositories`: 50 → **25** + +**New Configuration:** + +You can now configure page sizes in `app-config.yaml` to customize GitHub API resource consumption: + +**For `githubOrg` provider (users and teams):** + +```yaml +catalog: + providers: + githubOrg: + - id: production + githubUrl: https://github.com + orgs: ['your-org'] + schedule: + frequency: { minutes: 30 } + timeout: { minutes: 3 } + # Optional: Customize page sizes (defaults shown below) + pageSizes: + teams: 25 # Default: 25 + teamMembers: 50 # Default: 50 + organizationMembers: 50 # Default: 50 + repositories: 25 # Default: 25 +``` + +**For `github` provider (repositories):** + +```yaml +catalog: + providers: + github: + myorg: + organization: 'your-org' + catalogPath: '/catalog-info.yaml' + schedule: + frequency: { minutes: 30 } + timeout: { minutes: 3 } + # Optional: Customize page sizes (defaults shown below) + pageSizes: + repositories: 25 # Default: 25 +``` + +**Breaking Changes:** + +The default page sizes have been reduced by 50% to prevent `RESOURCE_LIMITS_EXCEEDED` errors with large organizations. This may result in: + +- ✅ **More stable syncs** for large organizations (200+ teams) +- ⚠️ **Slightly more API calls** due to additional pagination +- ⚠️ **Slightly slower sync times** (typically 10-20% slower) + +If you need the previous behavior, you can restore the old values in your configuration: + +```yaml +pageSizes: + teams: 50 + teamMembers: 100 + organizationMembers: 100 + repositories: 50 +``` + +**Benefits:** + +- Prevents `RESOURCE_LIMITS_EXCEEDED` errors for large GitHub organizations (200+ teams) +- Configurable per provider instance +- No performance impact for smaller organizations +- All data still synced through pagination + +Resolves GitHub issue #31437 diff --git a/docs/integrations/github/discovery.md b/docs/integrations/github/discovery.md index 4a40b25361..8546e297c6 100644 --- a/docs/integrations/github/discovery.md +++ b/docs/integrations/github/discovery.md @@ -308,6 +308,26 @@ If you do so, `default` will be used as provider ID. The amount of time that should pass before the first invocation happens. - **`scope`** _(optional)_: `'global'` or `'local'`. Sets the scope of concurrency control. +- **`pageSizes`** _(optional)_: + Configure page sizes for GitHub GraphQL API queries. This can help prevent `RESOURCE_LIMITS_EXCEEDED` errors with large organizations. + - **`repositories`** _(optional)_: + Number of repositories to fetch per page. Defaults to `25`. + +Example with page sizes configuration: + +```yaml +catalog: + providers: + github: + myOrganization: + organization: 'my-large-org' + catalogPath: '/catalog-info.yaml' + schedule: + frequency: { minutes: 30 } + timeout: { minutes: 3 } + pageSizes: + repositories: 15 # Reduce if hitting API limits +``` ## GitHub API Rate Limits diff --git a/docs/integrations/github/org.md b/docs/integrations/github/org.md index 473d408b0b..9962ae641e 100644 --- a/docs/integrations/github/org.md +++ b/docs/integrations/github/org.md @@ -94,6 +94,37 @@ Directly under the `githubOrg` is a list of configurations, each entry is a stru - `githubUrl`: The target that this provider should consume - `orgs` (optional): The list of the GitHub orgs to consume. If you only list a single org the generated group entities will use the `default` namespace, otherwise they will use the org name as the namespace. By default the provider will consume all accessible orgs on the given GitHub instance (support for GitHub App integration only). - `schedule`: The refresh schedule to use, matches the structure of [`SchedulerServiceTaskScheduleDefinitionConfig`](https://backstage.io/docs/reference/backend-plugin-api.schedulerservicetaskscheduledefinitionconfig/) +- `pageSizes` (optional): Configure page sizes for GitHub GraphQL API queries to prevent `RESOURCE_LIMITS_EXCEEDED` errors with large organizations. See [Page Sizes Configuration](#page-sizes-configuration) below for details. + +### Page Sizes Configuration + +For large GitHub organizations (200+ teams), you may encounter `RESOURCE_LIMITS_EXCEEDED` errors due to GitHub's GraphQL API resource limits. You can configure page sizes to reduce the number of records fetched per API request: + +```yaml title="app-config.yaml" +catalog: + providers: + githubOrg: + - id: production + githubUrl: https://github.com + orgs: ['large-org'] + schedule: + frequency: { hours: 1 } + timeout: { minutes: 50 } + pageSizes: + teams: 25 # Default: 25 + teamMembers: 50 # Default: 50 + organizationMembers: 50 # Default: 50 + repositories: 25 # Default: 25 +``` + +**Configuration Options:** + +- `teams`: Number of teams to fetch per page when querying organization teams (default: 25) +- `teamMembers`: Number of team members to fetch per page when querying team members (default: 50) +- `organizationMembers`: Number of organization members to fetch per page (default: 50) +- `repositories`: Number of repositories to fetch per page (default: 25) + +**Note:** Reducing page sizes will result in more API calls and slightly longer sync times, but will prevent resource limit errors for large organizations. ### Events Support diff --git a/plugins/catalog-backend-module-github-org/src/module.test.ts b/plugins/catalog-backend-module-github-org/src/module.test.ts index ba69f3c842..da48f013cb 100644 --- a/plugins/catalog-backend-module-github-org/src/module.test.ts +++ b/plugins/catalog-backend-module-github-org/src/module.test.ts @@ -73,4 +73,106 @@ describe('catalogModuleGithubOrgEntityProvider', () => { ); expect(runner).not.toHaveBeenCalled(); }); + + it('should register provider with custom page sizes', async () => { + let addedProviders: Array | undefined; + + const extensionPoint = { + addEntityProvider: (...providers: any) => { + addedProviders = providers; + }, + }; + const runner = jest.fn(); + const scheduler = mockServices.scheduler.mock({ + createScheduledTaskRunner() { + return { run: runner }; + }, + }); + + const config = { + catalog: { + providers: { + githubOrg: [ + { + id: 'default', + githubUrl: 'https://github.com', + orgs: ['backstage'], + schedule: { + frequency: 'P1M', + timeout: 'PT3M', + }, + pageSizes: { + teams: 10, + teamMembers: 25, + organizationMembers: 30, + repositories: 15, + }, + }, + ], + }, + }, + }; + + await startTestBackend({ + extensionPoints: [[catalogProcessingExtensionPoint, extensionPoint]], + features: [ + catalogModuleGithubOrgEntityProvider, + mockServices.rootConfig.factory({ data: config }), + scheduler.factory, + ], + }); + + expect(addedProviders?.length).toEqual(1); + expect(addedProviders![0].getProviderName()).toEqual( + 'GithubMultiOrgEntityProvider:default', + ); + }); + + it('should register provider without page sizes configuration', async () => { + let addedProviders: Array | undefined; + + const extensionPoint = { + addEntityProvider: (...providers: any) => { + addedProviders = providers; + }, + }; + const runner = jest.fn(); + const scheduler = mockServices.scheduler.mock({ + createScheduledTaskRunner() { + return { run: runner }; + }, + }); + + const config = { + catalog: { + providers: { + githubOrg: [ + { + id: 'default', + githubUrl: 'https://github.com', + orgs: ['backstage'], + schedule: { + frequency: 'P1M', + timeout: 'PT3M', + }, + }, + ], + }, + }, + }; + + await startTestBackend({ + extensionPoints: [[catalogProcessingExtensionPoint, extensionPoint]], + features: [ + catalogModuleGithubOrgEntityProvider, + mockServices.rootConfig.factory({ data: config }), + scheduler.factory, + ], + }); + + expect(addedProviders?.length).toEqual(1); + expect(addedProviders![0].getProviderName()).toEqual( + 'GithubMultiOrgEntityProvider:default', + ); + }); }); diff --git a/plugins/catalog-backend-module-github-org/src/module.ts b/plugins/catalog-backend-module-github-org/src/module.ts index adb6f6c42d..b9cd227abf 100644 --- a/plugins/catalog-backend-module-github-org/src/module.ts +++ b/plugins/catalog-backend-module-github-org/src/module.ts @@ -120,6 +120,7 @@ export const catalogModuleGithubOrgEntityProvider = createBackendModule({ teamTransformer, alwaysUseDefaultNamespace: definitions.length === 1 && definition.orgs?.length === 1, + pageSizes: definition.pageSizes, }), ); } @@ -133,6 +134,12 @@ function readDefinitionsFromConfig(rootConfig: Config): Array<{ githubUrl: string; orgs?: string[]; schedule: SchedulerServiceTaskScheduleDefinition; + pageSizes?: { + teams?: number; + teamMembers?: number; + organizationMembers?: number; + repositories?: number; + }; }> { const baseKey = 'catalog.providers.githubOrg'; const baseConfig = rootConfig.getOptional(baseKey); @@ -151,5 +158,15 @@ function readDefinitionsFromConfig(rootConfig: Config): Array<{ schedule: readSchedulerServiceTaskScheduleDefinitionFromConfig( c.getConfig('schedule'), ), + pageSizes: c.has('pageSizes') + ? { + teams: c.getOptionalNumber('pageSizes.teams'), + teamMembers: c.getOptionalNumber('pageSizes.teamMembers'), + organizationMembers: c.getOptionalNumber( + 'pageSizes.organizationMembers', + ), + repositories: c.getOptionalNumber('pageSizes.repositories'), + } + : undefined, })); } diff --git a/plugins/catalog-backend-module-github/config.d.ts b/plugins/catalog-backend-module-github/config.d.ts index 1865df7753..d795af2774 100644 --- a/plugins/catalog-backend-module-github/config.d.ts +++ b/plugins/catalog-backend-module-github/config.d.ts @@ -131,6 +131,18 @@ export interface Config { * (Optional) TaskScheduleDefinition for the refresh. */ schedule?: SchedulerServiceTaskScheduleDefinitionConfig; + + /** + * (Optional) Page sizes for GitHub GraphQL API queries. + * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors with large orgs. + */ + pageSizes?: { + /** + * (Optional) Number of repositories to fetch per page when querying repositories. + * Default: `25`. + */ + repositories?: number; + }; } | { [name: string]: { @@ -209,6 +221,18 @@ export interface Config { * (Optional) TaskScheduleDefinition for the refresh. */ schedule?: SchedulerServiceTaskScheduleDefinitionConfig; + + /** + * (Optional) Page sizes for GitHub GraphQL API queries. + * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors with large orgs. + */ + pageSizes?: { + /** + * (Optional) Number of repositories to fetch per page when querying repositories. + * Default: `25`. + */ + repositories?: number; + }; }; }; @@ -244,6 +268,33 @@ export interface Config { * The refresh schedule to use. */ schedule: SchedulerServiceTaskScheduleDefinitionConfig; + + /** + * (Optional) Page sizes for GitHub GraphQL API queries. + * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors with large orgs. + */ + pageSizes?: { + /** + * (Optional) Number of teams to fetch per page when querying organization teams. + * Default: `25`. + */ + teams?: number; + /** + * (Optional) Number of team members to fetch per page when querying team members. + * Default: `50`. + */ + teamMembers?: number; + /** + * (Optional) Number of organization members to fetch per page when querying org members. + * Default: `50`. + */ + organizationMembers?: number; + /** + * (Optional) Number of repositories to fetch per page when querying repositories. + * Default: `25`. + */ + repositories?: number; + }; } | Array<{ /** @@ -273,6 +324,33 @@ export interface Config { * The refresh schedule to use. */ schedule: SchedulerServiceTaskScheduleDefinitionConfig; + + /** + * (Optional) Page sizes for GitHub GraphQL API queries. + * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors with large orgs. + */ + pageSizes?: { + /** + * (Optional) Number of teams to fetch per page when querying organization teams. + * Default: `25`. + */ + teams?: number; + /** + * (Optional) Number of team members to fetch per page when querying team members. + * Default: `50`. + */ + teamMembers?: number; + /** + * (Optional) Number of organization members to fetch per page when querying org members. + * Default: `50`. + */ + organizationMembers?: number; + /** + * (Optional) Number of repositories to fetch per page when querying repositories. + * Default: `25`. + */ + repositories?: number; + }; }>; }; }; diff --git a/plugins/catalog-backend-module-github/report.api.md b/plugins/catalog-backend-module-github/report.api.md index 691cf271b4..ca23ff27c9 100644 --- a/plugins/catalog-backend-module-github/report.api.md +++ b/plugins/catalog-backend-module-github/report.api.md @@ -27,6 +27,9 @@ import { ScmIntegrationRegistry } from '@backstage/integration'; import { ScmLocationAnalyzer } from '@backstage/plugin-catalog-node'; import { UserEntity } from '@backstage/catalog-model'; +// @public +export const DEFAULT_PAGE_SIZES: GithubPageSizes; + // @public export const defaultOrganizationTeamTransformer: TeamTransformer; @@ -150,6 +153,7 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { userTransformer?: UserTransformer; teamTransformer?: TeamTransformer; alwaysUseDefaultNamespace?: boolean; + pageSizes?: Partial; }); connect(connection: EntityProviderConnection): Promise; // (undocumented) @@ -170,6 +174,7 @@ export interface GithubMultiOrgEntityProviderOptions { id: string; logger: LoggerService; orgs?: string[]; + pageSizes?: Partial; schedule?: 'manual' | SchedulerServiceTaskRunner; teamTransformer?: TeamTransformer; userTransformer?: UserTransformer; @@ -276,6 +281,14 @@ export class GithubOrgReaderProcessor implements CatalogProcessor { ): Promise; } +// @public +export type GithubPageSizes = { + teams: number; + teamMembers: number; + organizationMembers: number; + repositories: number; +}; + // @public export type GithubTeam = { slug: string; diff --git a/plugins/catalog-backend-module-github/src/index.ts b/plugins/catalog-backend-module-github/src/index.ts index 539e4ae14f..6f14bb25ef 100644 --- a/plugins/catalog-backend-module-github/src/index.ts +++ b/plugins/catalog-backend-module-github/src/index.ts @@ -40,6 +40,8 @@ export { type TeamTransformer, defaultOrganizationTeamTransformer, type TransformerContext, + type GithubPageSizes, + DEFAULT_PAGE_SIZES, } from './lib'; export * from './deprecated'; diff --git a/plugins/catalog-backend-module-github/src/lib/github.test.ts b/plugins/catalog-backend-module-github/src/lib/github.test.ts index 09eeba3cc0..f8a4fd6201 100644 --- a/plugins/catalog-backend-module-github/src/lib/github.test.ts +++ b/plugins/catalog-backend-module-github/src/lib/github.test.ts @@ -863,4 +863,144 @@ describe('github', () => { }); }); }); + + describe('Page sizes configuration', () => { + const org = 'my-org'; + + it('uses custom page sizes for getOrganizationTeams', async () => { + server.use( + graphqlMsw.query('teams', ({ variables }) => { + expect(variables.teamsPageSize).toBe(10); + expect(variables.membersPageSize).toBe(20); + return HttpResponse.json({ + data: { + organization: { + teams: { + pageInfo: { hasNextPage: false, endCursor: null }, + nodes: [ + { + slug: 'team1', + combinedSlug: 'my-org/team1', + name: 'Team 1', + description: 'desc', + avatarUrl: '', + editTeamUrl: '', + parentTeam: null, + members: { + pageInfo: { hasNextPage: false }, + nodes: [{ login: 'user1' }], + }, + }, + ], + }, + }, + }, + }); + }), + ); + + await getOrganizationTeams(graphql as any, org, undefined, { + teams: 10, + teamMembers: 20, + organizationMembers: 20, + repositories: 10, + }); + }); + + it('uses custom page sizes for getOrganizationUsers', async () => { + server.use( + graphqlMsw.query('users', ({ variables }) => { + expect(variables.pageSize).toBe(30); + return HttpResponse.json({ + data: { + organization: { + membersWithRole: { + pageInfo: { hasNextPage: false, endCursor: null }, + nodes: [ + { + login: 'user1', + name: 'User 1', + bio: '', + avatarUrl: '', + email: 'user1@example.com', + organizationVerifiedDomainEmails: [], + }, + ], + }, + }, + }, + }); + }), + ); + + await getOrganizationUsers(graphql as any, org, 'token', undefined, { + teams: 10, + teamMembers: 20, + organizationMembers: 30, + repositories: 10, + }); + }); + + it('uses custom page sizes for getOrganizationRepositories', async () => { + server.use( + graphqlMsw.query('repositories', ({ variables }) => { + expect(variables.repositoriesPageSize).toBe(15); + return HttpResponse.json({ + data: { + repositoryOwner: { + repositories: { + pageInfo: { hasNextPage: false, endCursor: null }, + nodes: [ + { + name: 'repo1', + url: 'https://github.com/my-org/repo1', + isArchived: false, + isFork: false, + visibility: 'public', + defaultBranchRef: { name: 'main' }, + catalogInfoFile: null, + repositoryTopics: { nodes: [] }, + }, + ], + }, + }, + }, + }); + }), + ); + + await getOrganizationRepositories( + graphql as any, + org, + '/catalog-info.yaml', + { + teams: 10, + teamMembers: 20, + organizationMembers: 30, + repositories: 15, + }, + ); + }); + + it('uses default page sizes when not specified', async () => { + server.use( + graphqlMsw.query('teams', ({ variables }) => { + expect(variables.teamsPageSize).toBe(25); + expect(variables.membersPageSize).toBe(50); + return HttpResponse.json({ + data: { + organization: { + teams: { + pageInfo: { hasNextPage: false, endCursor: null }, + nodes: [], + }, + }, + }, + }); + }), + ); + + await getOrganizationTeams(graphql as any, org); + }); + }); }); diff --git a/plugins/catalog-backend-module-github/src/lib/github.ts b/plugins/catalog-backend-module-github/src/lib/github.ts index 5d51dd35cd..a3a88b1e97 100644 --- a/plugins/catalog-backend-module-github/src/lib/github.ts +++ b/plugins/catalog-backend-module-github/src/lib/github.ts @@ -30,6 +30,48 @@ import { DeferredEntity } from '@backstage/plugin-catalog-node'; import { Octokit } from '@octokit/core'; import { LoggerService } from '@backstage/backend-plugin-api'; import { throttling } from '@octokit/plugin-throttling'; + +/** + * Configuration for GitHub GraphQL API page sizes. + * + * @public + */ +export type GithubPageSizes = { + /** + * Number of teams to fetch per page when querying organization teams. + * Default: 25 + */ + teams: number; + /** + * Number of team members to fetch per page when querying team members. + * Default: 50 + */ + teamMembers: number; + /** + * Number of organization members to fetch per page when querying org members. + * Default: 50 + */ + organizationMembers: number; + /** + * Number of repositories to fetch per page when querying repositories. + * Default: 25 + */ + repositories: number; +}; + +/** + * Default page sizes for GitHub GraphQL API queries. + * These values are reduced to prevent RESOURCE_LIMITS_EXCEEDED errors with large organizations. + * + * @public + */ +export const DEFAULT_PAGE_SIZES: GithubPageSizes = { + teams: 25, + teamMembers: 50, + organizationMembers: 50, + repositories: 25, +}; + // Graphql types export type QueryResponse = { @@ -136,17 +178,21 @@ export type Connection = { * * @param client - An octokit graphql client * @param org - The slug of the org to read + * @param tokenType - The type of GitHub credential + * @param userTransformer - Optional transformer for user entities + * @param pageSizes - Optional page sizes configuration */ export async function getOrganizationUsers( client: typeof graphql, org: string, tokenType: GithubCredentialType, userTransformer: UserTransformer = defaultUserTransformer, + pageSizes: GithubPageSizes = DEFAULT_PAGE_SIZES, ): Promise<{ users: Entity[] }> { const query = ` - query users($org: String!, $email: Boolean!, $cursor: String) { + query users($org: String!, $email: Boolean!, $cursor: String, $pageSize: Int!) { organization(login: $org) { - membersWithRole(first: 100, after: $cursor) { + membersWithRole(first: $pageSize, after: $cursor) { pageInfo { hasNextPage, endCursor } nodes { avatarUrl, @@ -172,6 +218,7 @@ export async function getOrganizationUsers( { org, email: tokenType === 'token', + pageSize: pageSizes.organizationMembers, }, ); @@ -185,18 +232,21 @@ export async function getOrganizationUsers( * * @param client - An octokit graphql client * @param org - The slug of the org to read + * @param teamTransformer - Optional transformer for team entities + * @param pageSizes - Optional page sizes configuration */ export async function getOrganizationTeams( client: typeof graphql, org: string, teamTransformer: TeamTransformer = defaultOrganizationTeamTransformer, + pageSizes: GithubPageSizes = DEFAULT_PAGE_SIZES, ): Promise<{ teams: Entity[]; }> { const query = ` - query teams($org: String!, $cursor: String) { + query teams($org: String!, $cursor: String, $teamsPageSize: Int!, $membersPageSize: Int!) { organization(login: $org) { - teams(first: 50, after: $cursor) { + teams(first: $teamsPageSize, after: $cursor) { pageInfo { hasNextPage, endCursor } nodes { slug @@ -206,7 +256,7 @@ export async function getOrganizationTeams( avatarUrl editTeamUrl parentTeam { slug } - members(first: 100, membership: IMMEDIATE) { + members(first: $membersPageSize, membership: IMMEDIATE) { pageInfo { hasNextPage } nodes { avatarUrl, @@ -234,9 +284,14 @@ export async function getOrganizationTeams( memberNames.push(user); } } else { - // There were more than a hundred immediate members - run the slow + // There were more immediate members than page size - run the slow // path of fetching them explicitly - const { members } = await getTeamMembers(ctx.client, ctx.org, item.slug); + const { members } = await getTeamMembers( + ctx.client, + ctx.org, + item.slug, + pageSizes, + ); for (const userLogin of members) { memberNames.push(userLogin); } @@ -256,7 +311,11 @@ export async function getOrganizationTeams( org, r => r.organization?.teams, materialisedTeams, - { org }, + { + org, + teamsPageSize: pageSizes.teams, + membersPageSize: pageSizes.teamMembers, + }, ); return { teams }; @@ -267,13 +326,14 @@ export async function getOrganizationTeamsFromUsers( org: string, userLogins: string[], teamTransformer: TeamTransformer = defaultOrganizationTeamTransformer, + pageSizes: GithubPageSizes = DEFAULT_PAGE_SIZES, ): Promise<{ teams: Entity[]; }> { const query = ` - query teams($org: String!, $cursor: String, $userLogins: [String!] = "") { + query teams($org: String!, $cursor: String, $userLogins: [String!] = "", $teamsPageSize: Int!, $membersPageSize: Int!) { organization(login: $org) { - teams(first: 100, after: $cursor, userLogins: $userLogins) { + teams(first: $teamsPageSize, after: $cursor, userLogins: $userLogins) { pageInfo { hasNextPage endCursor @@ -288,7 +348,7 @@ export async function getOrganizationTeamsFromUsers( parentTeam { slug } - members(first: 100, membership: IMMEDIATE) { + members(first: $membersPageSize, membership: IMMEDIATE) { pageInfo { hasNextPage } @@ -318,9 +378,14 @@ export async function getOrganizationTeamsFromUsers( memberNames.push(user); } } else { - // There were more than a hundred immediate members - run the slow + // There were more immediate members than page size - run the slow // path of fetching them explicitly - const { members } = await getTeamMembers(ctx.client, ctx.org, item.slug); + const { members } = await getTeamMembers( + ctx.client, + ctx.org, + item.slug, + pageSizes, + ); for (const userLogin of members) { memberNames.push(userLogin); } @@ -340,7 +405,12 @@ export async function getOrganizationTeamsFromUsers( org, r => r.organization?.teams, materialisedTeams, - { org, userLogins }, + { + org, + userLogins, + teamsPageSize: pageSizes.teams, + membersPageSize: pageSizes.teamMembers, + }, ); return { teams }; @@ -351,11 +421,12 @@ export async function getOrganizationTeamsForUser( org: string, userLogin: string, teamTransformer: TeamTransformer, + pageSizes: GithubPageSizes = DEFAULT_PAGE_SIZES, ): Promise<{ teams: Entity[] }> { const query = ` - query teams($org: String!, $cursor: String, $userLogins: [String!] = "") { + query teams($org: String!, $cursor: String, $userLogins: [String!] = "", $teamsPageSize: Int!) { organization(login: $org) { - teams(first: 100, after: $cursor, userLogins: $userLogins) { + teams(first: $teamsPageSize, after: $cursor, userLogins: $userLogins) { pageInfo { hasNextPage endCursor @@ -393,7 +464,7 @@ export async function getOrganizationTeamsForUser( org, r => r.organization?.teams, materialisedTeams, - { org, userLogins: [userLogin] }, + { org, userLogins: [userLogin], teamsPageSize: pageSizes.teams }, ); return { teams }; @@ -432,11 +503,12 @@ export async function getOrganizationTeam( org: string, teamSlug: string, teamTransformer: TeamTransformer = defaultOrganizationTeamTransformer, + pageSizes: GithubPageSizes = DEFAULT_PAGE_SIZES, ): Promise<{ team: Entity; }> { const query = ` - query teams($org: String!, $teamSlug: String!) { + query teams($org: String!, $teamSlug: String!, $membersPageSize: Int!) { organization(login: $org) { team(slug:$teamSlug) { slug @@ -446,7 +518,7 @@ export async function getOrganizationTeam( avatarUrl editTeamUrl parentTeam { slug } - members(first: 100, membership: IMMEDIATE) { + members(first: $membersPageSize, membership: IMMEDIATE) { pageInfo { hasNextPage } nodes { login } } @@ -466,9 +538,14 @@ export async function getOrganizationTeam( memberNames.push(user); } } else { - // There were more than a hundred immediate members - run the slow + // There were more immediate members than page size - run the slow // path of fetching them explicitly - const { members } = await getTeamMembers(ctx.client, ctx.org, item.slug); + const { members } = await getTeamMembers( + ctx.client, + ctx.org, + item.slug, + pageSizes, + ); for (const userLogin of members) { memberNames.push(userLogin); } @@ -485,6 +562,7 @@ export async function getOrganizationTeam( const response: QueryResponse = await client(query, { org, teamSlug, + membersPageSize: pageSizes.teamMembers, }); if (!response.organization?.team) @@ -505,6 +583,7 @@ export async function getOrganizationRepositories( client: typeof graphql, org: string, catalogPath: string, + pageSizes: GithubPageSizes = DEFAULT_PAGE_SIZES, ): Promise<{ repositories: RepositoryResponse[] }> { let relativeCatalogPathRef: string; // We must strip the leading slash or the query for objects does not work @@ -515,10 +594,10 @@ export async function getOrganizationRepositories( } const catalogPathRef = `HEAD:${relativeCatalogPathRef}`; const query = ` - query repositories($org: String!, $catalogPathRef: String!, $cursor: String) { + query repositories($org: String!, $catalogPathRef: String!, $cursor: String, $repositoriesPageSize: Int!) { repositoryOwner(login: $org) { login - repositories(first: 50, after: $cursor) { + repositories(first: $repositoriesPageSize, after: $cursor) { nodes { name catalogInfoFile: object(expression: $catalogPathRef) { @@ -559,7 +638,7 @@ export async function getOrganizationRepositories( org, r => r.repositoryOwner?.repositories, async x => x, - { org, catalogPathRef }, + { org, catalogPathRef, repositoriesPageSize: pageSizes.repositories }, ); return { repositories }; @@ -621,24 +700,26 @@ export async function getOrganizationRepository( } /** - * Gets all the users out of a Github organization. + * Gets all the users out of a Github organization team. * * Note that the users will not have their memberships filled in. * * @param client - An octokit graphql client * @param org - The slug of the org to read * @param teamSlug - The slug of the team to read + * @param pageSizes - Optional page sizes configuration */ export async function getTeamMembers( client: typeof graphql, org: string, teamSlug: string, + pageSizes: GithubPageSizes = DEFAULT_PAGE_SIZES, ): Promise<{ members: GithubUser[] }> { const query = ` - query members($org: String!, $teamSlug: String!, $cursor: String) { + query members($org: String!, $teamSlug: String!, $cursor: String, $membersPageSize: Int!) { organization(login: $org) { team(slug: $teamSlug) { - members(first: 100, after: $cursor, membership: IMMEDIATE) { + members(first: $membersPageSize, after: $cursor, membership: IMMEDIATE) { pageInfo { hasNextPage, endCursor } nodes { login } } @@ -652,7 +733,7 @@ export async function getTeamMembers( org, r => r.organization?.team?.members, async user => user, - { org, teamSlug }, + { org, teamSlug, membersPageSize: pageSizes.teamMembers }, ); return { members }; diff --git a/plugins/catalog-backend-module-github/src/lib/index.ts b/plugins/catalog-backend-module-github/src/lib/index.ts index f5da0f1c2b..ac73430d33 100644 --- a/plugins/catalog-backend-module-github/src/lib/index.ts +++ b/plugins/catalog-backend-module-github/src/lib/index.ts @@ -22,6 +22,8 @@ export { getOrganizationUsers, type GithubUser, type GithubTeam, + type GithubPageSizes, + DEFAULT_PAGE_SIZES, } from './github'; export { type UserTransformer, diff --git a/plugins/catalog-backend-module-github/src/providers/GithubEntityProvider.ts b/plugins/catalog-backend-module-github/src/providers/GithubEntityProvider.ts index 40ccf2aaaa..cbb719a04e 100644 --- a/plugins/catalog-backend-module-github/src/providers/GithubEntityProvider.ts +++ b/plugins/catalog-backend-module-github/src/providers/GithubEntityProvider.ts @@ -42,6 +42,8 @@ import { getOrganizationRepositories, getOrganizationRepository, RepositoryResponse, + GithubPageSizes, + DEFAULT_PAGE_SIZES, } from '../lib/github'; import { satisfiesForkFilter, @@ -262,8 +264,18 @@ export class GithubEntityProvider implements EntityProvider, EventSubscriber { for (const organization of organizations) { const client = await this.createGraphqlClient(organization); + const pageSizes: GithubPageSizes = { + ...DEFAULT_PAGE_SIZES, + ...this.config.pageSizes, + }; + const { repositories: repositoriesFromGithub } = - await getOrganizationRepositories(client, organization, catalogPath); + await getOrganizationRepositories( + client, + organization, + catalogPath, + pageSizes, + ); repositories = repositories.concat( repositoriesFromGithub.map(r => this.createRepoFromGithubResponse(r, organization), diff --git a/plugins/catalog-backend-module-github/src/providers/GithubEntityProviderConfig.test.ts b/plugins/catalog-backend-module-github/src/providers/GithubEntityProviderConfig.test.ts index 3b1743e404..b0a8dd7654 100644 --- a/plugins/catalog-backend-module-github/src/providers/GithubEntityProviderConfig.test.ts +++ b/plugins/catalog-backend-module-github/src/providers/GithubEntityProviderConfig.test.ts @@ -425,4 +425,68 @@ describe('readProviderConfigs', () => { expect(() => readProviderConfigs(config)).toThrow(); }); + + it('reads page sizes configuration', () => { + const config = new ConfigReader({ + catalog: { + providers: { + github: { + organization: 'test-org', + pageSizes: { + repositories: 10, + }, + }, + }, + }, + }); + const providerConfigs = readProviderConfigs(config); + + expect(providerConfigs).toHaveLength(1); + expect(providerConfigs[0].pageSizes).toEqual({ + repositories: 10, + }); + }); + + it('handles missing page sizes configuration', () => { + const config = new ConfigReader({ + catalog: { + providers: { + github: { + organization: 'test-org', + }, + }, + }, + }); + const providerConfigs = readProviderConfigs(config); + + expect(providerConfigs).toHaveLength(1); + expect(providerConfigs[0].pageSizes).toBeUndefined(); + }); + + it('reads multiple providers with different page sizes', () => { + const config = new ConfigReader({ + catalog: { + providers: { + github: { + providerWithPageSizes: { + organization: 'test-org1', + pageSizes: { + repositories: 15, + }, + }, + providerWithoutPageSizes: { + organization: 'test-org2', + }, + }, + }, + }, + }); + const providerConfigs = readProviderConfigs(config); + + expect(providerConfigs).toHaveLength(2); + expect(providerConfigs[0].pageSizes).toEqual({ + repositories: 15, + }); + expect(providerConfigs[1].pageSizes).toBeUndefined(); + }); }); diff --git a/plugins/catalog-backend-module-github/src/providers/GithubEntityProviderConfig.ts b/plugins/catalog-backend-module-github/src/providers/GithubEntityProviderConfig.ts index 64fac09f6e..8d3e19dbf2 100644 --- a/plugins/catalog-backend-module-github/src/providers/GithubEntityProviderConfig.ts +++ b/plugins/catalog-backend-module-github/src/providers/GithubEntityProviderConfig.ts @@ -48,6 +48,9 @@ export type GithubEntityProviderConfig = { }; validateLocationsExist: boolean; schedule?: SchedulerServiceTaskScheduleDefinition; + pageSizes?: { + repositories?: number; + }; }; export type GithubTopicFilters = { @@ -128,6 +131,12 @@ function readProviderConfig( ) : DEFAULT_GITHUB_ENTITY_PROVIDER_CONFIG_SCHEDULE; + const pageSizes = config.has('pageSizes') + ? { + repositories: config.getOptionalNumber('pageSizes.repositories'), + } + : undefined; + return { id, catalogPath, @@ -149,6 +158,7 @@ function readProviderConfig( }, schedule, validateLocationsExist, + pageSizes, }; } diff --git a/plugins/catalog-backend-module-github/src/providers/GithubMultiOrgEntityProvider.ts b/plugins/catalog-backend-module-github/src/providers/GithubMultiOrgEntityProvider.ts index a1c6a17209..bb0cc98c40 100644 --- a/plugins/catalog-backend-module-github/src/providers/GithubMultiOrgEntityProvider.ts +++ b/plugins/catalog-backend-module-github/src/providers/GithubMultiOrgEntityProvider.ts @@ -66,6 +66,8 @@ import { TeamTransformer, TransformerContext, UserTransformer, + GithubPageSizes, + DEFAULT_PAGE_SIZES, } from '../lib'; import { ANNOTATION_GITHUB_TEAM_SLUG, @@ -166,6 +168,12 @@ export interface GithubMultiOrgEntityProviderOptions { * By default, groups will be namespaced according to their GitHub org. */ teamTransformer?: TeamTransformer; + + /** + * Optionally configure page sizes for GitHub GraphQL API queries. + * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors with large orgs. + */ + pageSizes?: Partial; } type CreateDeltaOperation = (entities: Entity[]) => { @@ -212,6 +220,7 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { teamTransformer: options.teamTransformer, events: options.events, alwaysUseDefaultNamespace: options.alwaysUseDefaultNamespace, + pageSizes: options.pageSizes, }); provider.schedule(options.schedule); @@ -231,6 +240,7 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { userTransformer?: UserTransformer; teamTransformer?: TeamTransformer; alwaysUseDefaultNamespace?: boolean; + pageSizes?: Partial; }, ) {} @@ -239,6 +249,13 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { return `GithubMultiOrgEntityProvider:${this.options.id}`; } + private getPageSizes(): GithubPageSizes { + return { + ...DEFAULT_PAGE_SIZES, + ...this.options.pageSizes, + }; + } + /** {@inheritdoc @backstage/plugin-catalog-node#EntityProvider.connect} */ async connect(connection: EntityProviderConnection) { this.connection = connection; @@ -281,17 +298,21 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { logger.info(`Reading GitHub users and teams for org: ${org}`); + const pageSizes = this.getPageSizes(); + const { users } = await getOrganizationUsers( client, org, tokenType, this.options.userTransformer, + pageSizes, ); const { teams } = await getOrganizationTeams( client, org, this.defaultMultiOrgTeamTransformer.bind(this), + pageSizes, ); // Grab current users from `allUsersMap` if they already exist in our @@ -429,17 +450,21 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { headers, }); + const pageSizes = this.getPageSizes(); + const { users } = await getOrganizationUsers( client, org, tokenType, this.options.userTransformer, + pageSizes, ); const { teams } = await getOrganizationTeams( client, org, this.defaultMultiOrgTeamTransformer.bind(this), + pageSizes, ); if (users.length) { @@ -464,6 +489,7 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { u.metadata.name, ), this.defaultMultiOrgTeamTransformer.bind(this), + pageSizes, ); if (areGroupEntities(userTeams) && areUserEntities(users)) { @@ -548,6 +574,7 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { } if (updateMemberships) { + const pageSizes = this.getPageSizes(); for (const userOrg of userApplicableOrgs) { const { headers: orgHeaders } = await this.options.githubCredentialsProvider.getCredentials({ @@ -563,6 +590,7 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { userOrg, login, this.defaultMultiOrgTeamTransformer.bind(this), + pageSizes, ); if (isUserEntity(user) && areGroupEntities(teams)) { @@ -648,12 +676,14 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { headers, }); + const pageSizes = this.getPageSizes(); const teamSlug = event.team.slug; const { team } = await getOrganizationTeam( client, org, teamSlug, this.defaultMultiOrgTeamTransformer.bind(this), + pageSizes, ); const { users } = await getOrganizationUsers( @@ -661,6 +691,7 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { org, tokenType, this.options.userTransformer, + pageSizes, ); const usersFromChangedGroup = isGroupEntity(team) @@ -693,6 +724,7 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { u.metadata.name, ), this.defaultMultiOrgTeamTransformer.bind(this), + pageSizes, ); if (areGroupEntities(teams) && areUserEntities(usersToRebuild)) { @@ -761,12 +793,14 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { headers, }); + const pageSizes = this.getPageSizes(); const teamSlug = event.team.slug; const { team } = await getOrganizationTeam( client, org, teamSlug, this.defaultMultiOrgTeamTransformer.bind(this), + pageSizes, ); const userTransformer = @@ -806,6 +840,7 @@ export class GithubMultiOrgEntityProvider implements EntityProvider { userOrg, login, this.defaultMultiOrgTeamTransformer.bind(this), + pageSizes, ); if (areGroupEntities(teams)) { From 999d1c17479122f847ff9338c5bdd1bf792c48b1 Mon Sep 17 00:00:00 2001 From: abhishekbvs Date: Fri, 24 Oct 2025 00:32:24 +0530 Subject: [PATCH 2/5] Update docs and code based on review comments Signed-off-by: abhishekbvs --- .changeset/github-api-page-sizes.md | 83 ------------------- .changeset/pretty-kids-allow.md | 5 ++ .changeset/tough-sloths-spend.md | 5 ++ docs/integrations/github/discovery.md | 22 +---- docs/integrations/github/org.md | 38 ++------- .../src/module.test.ts | 1 - .../src/module.ts | 2 - .../catalog-backend-module-github/config.d.ts | 18 +--- .../src/index.ts | 2 - .../src/lib/github.test.ts | 2 +- .../src/lib/github.ts | 6 +- .../providers/GithubMultiOrgEntityProvider.ts | 2 +- 12 files changed, 32 insertions(+), 154 deletions(-) delete mode 100644 .changeset/github-api-page-sizes.md create mode 100644 .changeset/pretty-kids-allow.md create mode 100644 .changeset/tough-sloths-spend.md diff --git a/.changeset/github-api-page-sizes.md b/.changeset/github-api-page-sizes.md deleted file mode 100644 index 724adbd764..0000000000 --- a/.changeset/github-api-page-sizes.md +++ /dev/null @@ -1,83 +0,0 @@ ---- -'@backstage/plugin-catalog-backend-module-github': minor -'@backstage/plugin-catalog-backend-module-github-org': minor ---- - -Added configurable page sizes for GitHub GraphQL API queries to prevent `RESOURCE_LIMITS_EXCEEDED` errors with large GitHub organizations. - -**Default Values Changed:** - -To prevent `RESOURCE_LIMITS_EXCEEDED` errors by default, the page sizes have been reduced to 50% of previous values: - -- `teams`: 50 → **25** -- `teamMembers`: 100 → **50** -- `organizationMembers`: 100 → **50** -- `repositories`: 50 → **25** - -**New Configuration:** - -You can now configure page sizes in `app-config.yaml` to customize GitHub API resource consumption: - -**For `githubOrg` provider (users and teams):** - -```yaml -catalog: - providers: - githubOrg: - - id: production - githubUrl: https://github.com - orgs: ['your-org'] - schedule: - frequency: { minutes: 30 } - timeout: { minutes: 3 } - # Optional: Customize page sizes (defaults shown below) - pageSizes: - teams: 25 # Default: 25 - teamMembers: 50 # Default: 50 - organizationMembers: 50 # Default: 50 - repositories: 25 # Default: 25 -``` - -**For `github` provider (repositories):** - -```yaml -catalog: - providers: - github: - myorg: - organization: 'your-org' - catalogPath: '/catalog-info.yaml' - schedule: - frequency: { minutes: 30 } - timeout: { minutes: 3 } - # Optional: Customize page sizes (defaults shown below) - pageSizes: - repositories: 25 # Default: 25 -``` - -**Breaking Changes:** - -The default page sizes have been reduced by 50% to prevent `RESOURCE_LIMITS_EXCEEDED` errors with large organizations. This may result in: - -- ✅ **More stable syncs** for large organizations (200+ teams) -- ⚠️ **Slightly more API calls** due to additional pagination -- ⚠️ **Slightly slower sync times** (typically 10-20% slower) - -If you need the previous behavior, you can restore the old values in your configuration: - -```yaml -pageSizes: - teams: 50 - teamMembers: 100 - organizationMembers: 100 - repositories: 50 -``` - -**Benefits:** - -- Prevents `RESOURCE_LIMITS_EXCEEDED` errors for large GitHub organizations (200+ teams) -- Configurable per provider instance -- No performance impact for smaller organizations -- All data still synced through pagination - -Resolves GitHub issue #31437 diff --git a/.changeset/pretty-kids-allow.md b/.changeset/pretty-kids-allow.md new file mode 100644 index 0000000000..acb258c467 --- /dev/null +++ b/.changeset/pretty-kids-allow.md @@ -0,0 +1,5 @@ +--- +'@backstage/plugin-catalog-backend-module-github': patch +--- + +Added configurable pageSizes for GitHub GraphQL API queries to prevent RESOURCE_LIMITS_EXCEEDED errors with organizations with large number of teams, members and repositories. Default page sizes reduced by 50% to improve stability. diff --git a/.changeset/tough-sloths-spend.md b/.changeset/tough-sloths-spend.md new file mode 100644 index 0000000000..f88c3e14c3 --- /dev/null +++ b/.changeset/tough-sloths-spend.md @@ -0,0 +1,5 @@ +--- +'@backstage/plugin-catalog-backend-module-github-org': patch +--- + +Added pageSizes configuration schema to support configurable page sizes for GitHub GraphQL API queries. This enables the configuration to be defined in catalogModuleGithubOrgEntityProvider. diff --git a/docs/integrations/github/discovery.md b/docs/integrations/github/discovery.md index 8546e297c6..9724ef3177 100644 --- a/docs/integrations/github/discovery.md +++ b/docs/integrations/github/discovery.md @@ -210,6 +210,8 @@ catalog: filters: # optional filters branch: 'develop' # optional string repository: '.*' # optional Regex + pageSizes: + repositories: 25 wildcardProviderId: organization: 'new-org' # string catalogPath: '/groups/**/*.yaml' # this will search all folders for files that end in .yaml @@ -309,25 +311,9 @@ If you do so, `default` will be used as provider ID. - **`scope`** _(optional)_: `'global'` or `'local'`. Sets the scope of concurrency control. - **`pageSizes`** _(optional)_: - Configure page sizes for GitHub GraphQL API queries. This can help prevent `RESOURCE_LIMITS_EXCEEDED` errors with large organizations. + Configure page sizes for GitHub GraphQL API queries. This can help prevent `RESOURCE_LIMITS_EXCEEDED` errors. - **`repositories`** _(optional)_: - Number of repositories to fetch per page. Defaults to `25`. - -Example with page sizes configuration: - -```yaml -catalog: - providers: - github: - myOrganization: - organization: 'my-large-org' - catalogPath: '/catalog-info.yaml' - schedule: - frequency: { minutes: 30 } - timeout: { minutes: 3 } - pageSizes: - repositories: 15 # Reduce if hitting API limits -``` + Number of repositories to fetch per page. Defaults to `25`. Reduce this value if hitting API resource limits. ## GitHub API Rate Limits diff --git a/docs/integrations/github/org.md b/docs/integrations/github/org.md index 9962ae641e..6526183f2d 100644 --- a/docs/integrations/github/org.md +++ b/docs/integrations/github/org.md @@ -79,6 +79,10 @@ catalog: initialDelay: { seconds: 30 } frequency: { hours: 1 } timeout: { minutes: 50 } + pageSizes: + teams: 25 + teamMembers: 50 + organizationMembers: 50 - id: ghe githubUrl: https://ghe.mycompany.com orgs: ['internal-1', 'internal-2', 'internal-3'] @@ -94,37 +98,13 @@ Directly under the `githubOrg` is a list of configurations, each entry is a stru - `githubUrl`: The target that this provider should consume - `orgs` (optional): The list of the GitHub orgs to consume. If you only list a single org the generated group entities will use the `default` namespace, otherwise they will use the org name as the namespace. By default the provider will consume all accessible orgs on the given GitHub instance (support for GitHub App integration only). - `schedule`: The refresh schedule to use, matches the structure of [`SchedulerServiceTaskScheduleDefinitionConfig`](https://backstage.io/docs/reference/backend-plugin-api.schedulerservicetaskscheduledefinitionconfig/) -- `pageSizes` (optional): Configure page sizes for GitHub GraphQL API queries to prevent `RESOURCE_LIMITS_EXCEEDED` errors with large organizations. See [Page Sizes Configuration](#page-sizes-configuration) below for details. +- `pageSizes` (optional): Configure page sizes for GitHub GraphQL API queries to prevent `RESOURCE_LIMITS_EXCEEDED` errors. You can configure the following page sizes: -### Page Sizes Configuration + - `teams`: Number of teams to fetch per page when querying organization teams (default: 25) + - `teamMembers`: Number of team members to fetch per page when querying team members (default: 50) + - `organizationMembers`: Number of organization members to fetch per page (default: 50) -For large GitHub organizations (200+ teams), you may encounter `RESOURCE_LIMITS_EXCEEDED` errors due to GitHub's GraphQL API resource limits. You can configure page sizes to reduce the number of records fetched per API request: - -```yaml title="app-config.yaml" -catalog: - providers: - githubOrg: - - id: production - githubUrl: https://github.com - orgs: ['large-org'] - schedule: - frequency: { hours: 1 } - timeout: { minutes: 50 } - pageSizes: - teams: 25 # Default: 25 - teamMembers: 50 # Default: 50 - organizationMembers: 50 # Default: 50 - repositories: 25 # Default: 25 -``` - -**Configuration Options:** - -- `teams`: Number of teams to fetch per page when querying organization teams (default: 25) -- `teamMembers`: Number of team members to fetch per page when querying team members (default: 50) -- `organizationMembers`: Number of organization members to fetch per page (default: 50) -- `repositories`: Number of repositories to fetch per page (default: 25) - -**Note:** Reducing page sizes will result in more API calls and slightly longer sync times, but will prevent resource limit errors for large organizations. + Reducing page sizes will result in more API calls and slightly longer sync times, but will prevent API resource limits for organizations with large number of teams and members. ### Events Support diff --git a/plugins/catalog-backend-module-github-org/src/module.test.ts b/plugins/catalog-backend-module-github-org/src/module.test.ts index da48f013cb..1fcff306fe 100644 --- a/plugins/catalog-backend-module-github-org/src/module.test.ts +++ b/plugins/catalog-backend-module-github-org/src/module.test.ts @@ -105,7 +105,6 @@ describe('catalogModuleGithubOrgEntityProvider', () => { teams: 10, teamMembers: 25, organizationMembers: 30, - repositories: 15, }, }, ], diff --git a/plugins/catalog-backend-module-github-org/src/module.ts b/plugins/catalog-backend-module-github-org/src/module.ts index b9cd227abf..2d349cb873 100644 --- a/plugins/catalog-backend-module-github-org/src/module.ts +++ b/plugins/catalog-backend-module-github-org/src/module.ts @@ -138,7 +138,6 @@ function readDefinitionsFromConfig(rootConfig: Config): Array<{ teams?: number; teamMembers?: number; organizationMembers?: number; - repositories?: number; }; }> { const baseKey = 'catalog.providers.githubOrg'; @@ -165,7 +164,6 @@ function readDefinitionsFromConfig(rootConfig: Config): Array<{ organizationMembers: c.getOptionalNumber( 'pageSizes.organizationMembers', ), - repositories: c.getOptionalNumber('pageSizes.repositories'), } : undefined, })); diff --git a/plugins/catalog-backend-module-github/config.d.ts b/plugins/catalog-backend-module-github/config.d.ts index d795af2774..f3cbb15e06 100644 --- a/plugins/catalog-backend-module-github/config.d.ts +++ b/plugins/catalog-backend-module-github/config.d.ts @@ -134,7 +134,7 @@ export interface Config { /** * (Optional) Page sizes for GitHub GraphQL API queries. - * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors with large orgs. + * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors. */ pageSizes?: { /** @@ -224,7 +224,7 @@ export interface Config { /** * (Optional) Page sizes for GitHub GraphQL API queries. - * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors with large orgs. + * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors. */ pageSizes?: { /** @@ -271,7 +271,7 @@ export interface Config { /** * (Optional) Page sizes for GitHub GraphQL API queries. - * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors with large orgs. + * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors. */ pageSizes?: { /** @@ -289,11 +289,6 @@ export interface Config { * Default: `50`. */ organizationMembers?: number; - /** - * (Optional) Number of repositories to fetch per page when querying repositories. - * Default: `25`. - */ - repositories?: number; }; } | Array<{ @@ -327,7 +322,7 @@ export interface Config { /** * (Optional) Page sizes for GitHub GraphQL API queries. - * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors with large orgs. + * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors. */ pageSizes?: { /** @@ -345,11 +340,6 @@ export interface Config { * Default: `50`. */ organizationMembers?: number; - /** - * (Optional) Number of repositories to fetch per page when querying repositories. - * Default: `25`. - */ - repositories?: number; }; }>; }; diff --git a/plugins/catalog-backend-module-github/src/index.ts b/plugins/catalog-backend-module-github/src/index.ts index 6f14bb25ef..539e4ae14f 100644 --- a/plugins/catalog-backend-module-github/src/index.ts +++ b/plugins/catalog-backend-module-github/src/index.ts @@ -40,8 +40,6 @@ export { type TeamTransformer, defaultOrganizationTeamTransformer, type TransformerContext, - type GithubPageSizes, - DEFAULT_PAGE_SIZES, } from './lib'; export * from './deprecated'; diff --git a/plugins/catalog-backend-module-github/src/lib/github.test.ts b/plugins/catalog-backend-module-github/src/lib/github.test.ts index f8a4fd6201..ca9b3eb692 100644 --- a/plugins/catalog-backend-module-github/src/lib/github.test.ts +++ b/plugins/catalog-backend-module-github/src/lib/github.test.ts @@ -910,7 +910,7 @@ describe('github', () => { it('uses custom page sizes for getOrganizationUsers', async () => { server.use( graphqlMsw.query('users', ({ variables }) => { - expect(variables.pageSize).toBe(30); + expect(variables.organizationMembersPageSize).toBe(30); return HttpResponse.json({ data: { organization: { diff --git a/plugins/catalog-backend-module-github/src/lib/github.ts b/plugins/catalog-backend-module-github/src/lib/github.ts index a3a88b1e97..625fb57bba 100644 --- a/plugins/catalog-backend-module-github/src/lib/github.ts +++ b/plugins/catalog-backend-module-github/src/lib/github.ts @@ -190,9 +190,9 @@ export async function getOrganizationUsers( pageSizes: GithubPageSizes = DEFAULT_PAGE_SIZES, ): Promise<{ users: Entity[] }> { const query = ` - query users($org: String!, $email: Boolean!, $cursor: String, $pageSize: Int!) { + query users($org: String!, $email: Boolean!, $cursor: String, $organizationMembersPageSize: Int!) { organization(login: $org) { - membersWithRole(first: $pageSize, after: $cursor) { + membersWithRole(first: $organizationMembersPageSize, after: $cursor) { pageInfo { hasNextPage, endCursor } nodes { avatarUrl, @@ -218,7 +218,7 @@ export async function getOrganizationUsers( { org, email: tokenType === 'token', - pageSize: pageSizes.organizationMembers, + organizationMembersPageSize: pageSizes.organizationMembers, }, ); diff --git a/plugins/catalog-backend-module-github/src/providers/GithubMultiOrgEntityProvider.ts b/plugins/catalog-backend-module-github/src/providers/GithubMultiOrgEntityProvider.ts index bb0cc98c40..839ca66e40 100644 --- a/plugins/catalog-backend-module-github/src/providers/GithubMultiOrgEntityProvider.ts +++ b/plugins/catalog-backend-module-github/src/providers/GithubMultiOrgEntityProvider.ts @@ -171,7 +171,7 @@ export interface GithubMultiOrgEntityProviderOptions { /** * Optionally configure page sizes for GitHub GraphQL API queries. - * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors with large orgs. + * Reduce these values if hitting RESOURCE_LIMITS_EXCEEDED errors. */ pageSizes?: Partial; } From 226b1802e3353321ed22f95de7b577af88b5b74e Mon Sep 17 00:00:00 2001 From: abhishekbvs Date: Fri, 24 Oct 2025 01:05:36 +0530 Subject: [PATCH 3/5] Update catalog-backend-module-github plugin report.api.md Signed-off-by: abhishekbvs --- plugins/catalog-backend-module-github/report.api.md | 3 --- plugins/catalog-backend-module-github/src/index.ts | 1 + 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/plugins/catalog-backend-module-github/report.api.md b/plugins/catalog-backend-module-github/report.api.md index ca23ff27c9..016bb6da65 100644 --- a/plugins/catalog-backend-module-github/report.api.md +++ b/plugins/catalog-backend-module-github/report.api.md @@ -27,9 +27,6 @@ import { ScmIntegrationRegistry } from '@backstage/integration'; import { ScmLocationAnalyzer } from '@backstage/plugin-catalog-node'; import { UserEntity } from '@backstage/catalog-model'; -// @public -export const DEFAULT_PAGE_SIZES: GithubPageSizes; - // @public export const defaultOrganizationTeamTransformer: TeamTransformer; diff --git a/plugins/catalog-backend-module-github/src/index.ts b/plugins/catalog-backend-module-github/src/index.ts index 539e4ae14f..6d62a0a64a 100644 --- a/plugins/catalog-backend-module-github/src/index.ts +++ b/plugins/catalog-backend-module-github/src/index.ts @@ -40,6 +40,7 @@ export { type TeamTransformer, defaultOrganizationTeamTransformer, type TransformerContext, + type GithubPageSizes, } from './lib'; export * from './deprecated'; From 68f7e2b0bc3569edda342e77050dcc5852b9cac1 Mon Sep 17 00:00:00 2001 From: Abhishek Bvs <32136294+abhishekbvs@users.noreply.github.com> Date: Sat, 25 Oct 2025 15:13:17 +0530 Subject: [PATCH 4/5] Update .changeset/pretty-kids-allow.md Co-authored-by: Andre Wanlin <67169551+awanlin@users.noreply.github.com> Signed-off-by: Abhishek Bvs <32136294+abhishekbvs@users.noreply.github.com> --- .changeset/pretty-kids-allow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.changeset/pretty-kids-allow.md b/.changeset/pretty-kids-allow.md index acb258c467..6b3b11a895 100644 --- a/.changeset/pretty-kids-allow.md +++ b/.changeset/pretty-kids-allow.md @@ -2,4 +2,4 @@ '@backstage/plugin-catalog-backend-module-github': patch --- -Added configurable pageSizes for GitHub GraphQL API queries to prevent RESOURCE_LIMITS_EXCEEDED errors with organizations with large number of teams, members and repositories. Default page sizes reduced by 50% to improve stability. +Added configurable `pageSizes` for GitHub GraphQL API queries to prevent `RESOURCE_LIMITS_EXCEEDED` errors with organizations with large number of repositories. Please see the [GitHub Discovery documentation](https://backstage.io/docs/integrations/github/discovery#configuration) for new configuration options. From ce2ad7f7182fdfef9ecec888b2e5b3c5fef86b28 Mon Sep 17 00:00:00 2001 From: Abhishek Bvs <32136294+abhishekbvs@users.noreply.github.com> Date: Sat, 25 Oct 2025 15:13:35 +0530 Subject: [PATCH 5/5] Update .changeset/tough-sloths-spend.md Co-authored-by: Andre Wanlin <67169551+awanlin@users.noreply.github.com> Signed-off-by: Abhishek Bvs <32136294+abhishekbvs@users.noreply.github.com> --- .changeset/tough-sloths-spend.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.changeset/tough-sloths-spend.md b/.changeset/tough-sloths-spend.md index f88c3e14c3..ed6a4ab366 100644 --- a/.changeset/tough-sloths-spend.md +++ b/.changeset/tough-sloths-spend.md @@ -2,4 +2,4 @@ '@backstage/plugin-catalog-backend-module-github-org': patch --- -Added pageSizes configuration schema to support configurable page sizes for GitHub GraphQL API queries. This enables the configuration to be defined in catalogModuleGithubOrgEntityProvider. +Added configurable `pageSizes` for GitHub GraphQL API queries to prevent `RESOURCE_LIMITS_EXCEEDED` errors with organizations with large number of teams and members. Please see the [GitHub Org Data documentation](https://backstage.io/docs/integrations/github/org#configuration-details) for new configuration options.