diff --git a/.changeset/whole-dodos-matter.md b/.changeset/whole-dodos-matter.md new file mode 100644 index 0000000000..df65f33ed7 --- /dev/null +++ b/.changeset/whole-dodos-matter.md @@ -0,0 +1,20 @@ +--- +'@backstage/plugin-techdocs-node': minor +--- + +**BREAKING:** It's now possible to use the credentials from the `integrations.awsS3` config to authenticate with AWS S3. The new priority is: + +1. `aws.accounts` +2. `techdocs.publisher.awsS3.credentials` +3. `integrations.awsS3` +4. Default credential chain + +In case of multiple `integrations.awsS3` are present, the target integration is determined by the `accessKeyId` in `techdocs.publisher.awsS3.credentials` if provided. Otherwise, the default credential chain is used. + +This means that depending on your setup, this feature may break your existing setup. +In general: + +- if you are configuring `aws.accounts`, no action is required +- if you are configuring `techdocs.publisher.awsS3.credentials`, no action is required +- if you are configuring multiple integrations under `integrations.awsS3`, no action is required +- if you are configuring a single integration under `integrations.awsS3`, make sure that the integration has access to the bucket you are using for TechDocs diff --git a/docs/features/techdocs/using-cloud-storage.md b/docs/features/techdocs/using-cloud-storage.md index 42024ffbb2..2d372dc48f 100644 --- a/docs/features/techdocs/using-cloud-storage.md +++ b/docs/features/techdocs/using-cloud-storage.md @@ -172,15 +172,17 @@ TechDocs will publish documentation to this bucket and will fetch files from here to serve documentation in Backstage. Note that the bucket names are globally unique. -Set the config `techdocs.publisher.awsS3.bucketName` in your `app-config.yaml` -to the name of the bucket you just created. +Set the bucket name and region in your `app-config.yaml` to the name of the bucket you just created: ```yaml techdocs: publisher: type: 'awsS3' +/* highlight-add-start */ awsS3: bucketName: 'name-of-techdocs-storage-bucket' + region: 'us-east-1' +/* highlight-add-end */ ``` **3. Create minimal AWS IAM policies to manage TechDocs** @@ -266,7 +268,7 @@ environment automatically by defining appropriate IAM role with access to the bucket. Read more in the [official AWS documentation for using IAM roles](https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html#use-roles). -**4b. Authentication using app-config.yaml** +**4b. Authentication using app-config.yaml via aws.accounts** AWS credentials and region can be provided to the AWS SDK via `app-config.yaml`. If the configs below are present, they will be used over existing `AWS_*` @@ -290,7 +292,45 @@ aws: Refer to the [official AWS documentation for obtaining the credentials](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/setting-credentials-node.html). -**4c. Authentication using an assumed role** Users with multiple AWS accounts +**4c. Authentication using app-config.yaml via integrations.awsS3** + +If you already have an [AWS S3 integration](../../integrations/aws-s3/locations.md), you can use it to authenticate with AWS S3: + +```yaml +techdocs: + publisher: + type: 'awsS3' + awsS3: + bucketName: 'name-of-techdocs-storage-bucket' + region: 'eu-west-1' +integrations: + awsS3: + - accessKeyId: ${AWS_ACCESS_KEY_ID} + secretAccessKey: ${AWS_SECRET_ACCESS_KEY} +``` + +This will use the credentials from the integration to authenticate with AWS S3 and it does not require any additional configuration in the `app-config.yaml`. However, **if you have multiple S3 integrations**, you **must** specify the target integration by setting the `accessKeyId` in the `techdocs.publisher.awsS3.credentials` config: + +```yaml +techdocs: + publisher: + type: 'awsS3' + awsS3: + bucketName: 'name-of-techdocs-storage-bucket' + region: 'eu-west-1' +/* highlight-add-start */ + credentials: + accessKeyId: ${AWS_ACCESS_KEY_ID_1} +/* highlight-add-end */ +integrations: + awsS3: + - accessKeyId: ${AWS_ACCESS_KEY_ID_1} + secretAccessKey: ${AWS_SECRET_ACCESS_KEY_1} + - accessKeyId: ${AWS_ACCESS_KEY_ID_2} + secretAccessKey: ${AWS_SECRET_ACCESS_KEY_2} +``` + +**4d. Authentication using an assumed role** Users with multiple AWS accounts may want to use a role for S3 storage that is in a different AWS account. Using the `roleArn` parameter as seen below, you can instruct the TechDocs publisher to assume a role before accessing S3. diff --git a/plugins/techdocs-node/src/stages/publish/awsS3.test.ts b/plugins/techdocs-node/src/stages/publish/awsS3.test.ts index 8be8c05994..c1a31d4a08 100644 --- a/plugins/techdocs-node/src/stages/publish/awsS3.test.ts +++ b/plugins/techdocs-node/src/stages/publish/awsS3.test.ts @@ -280,25 +280,230 @@ describe('AwsS3Publish', () => { expect(getCredProviderMock).toHaveBeenCalledTimes(1); }); - it('should fall back to deprecated method of retrieving credentials', async () => { + it('should use aws.accounts over integrations.awsS3 if both are provided', async () => { + await jest.isolateModulesAsync(async () => { + jest.doMock('@aws-sdk/client-s3', () => ({ + ...jest.requireActual('@aws-sdk/client-s3'), + S3Client: jest.fn(), + })); + + const { S3Client: MockS3Client } = require('@aws-sdk/client-s3'); + const { AwsS3Publish: IsolatedAwsS3Publish } = require('./awsS3'); + const mockConfig = new ConfigReader({ + techdocs: { + publisher: { + type: 'awsS3', + awsS3: { + accountId: '111111111111', + bucketName: 'bucketName', + bucketRootPath: '/', + }, + }, + }, + integrations: { + awsS3: [ + { + accessKeyId: 'access-key-from-integrations', + secretAccessKey: 'secret-access-key-from-integrations', + }, + ], + }, + aws: { + accounts: [ + { + accountId: '111111111111', + accessKeyId: 'access-key-from-aws', + secretAccessKey: 'secret-access-key-from-aws', + }, + ], + }, + }); + + await IsolatedAwsS3Publish.fromConfig(mockConfig, logger); + expect(getCredProviderMock).toHaveBeenCalledTimes(0); + expect(MockS3Client).toHaveBeenCalledTimes(1); + await expect( + MockS3Client.mock.calls[0][0]!.credentialDefaultProvider!( + undefined!, + )(), + ).resolves.toEqual({ + accessKeyId: 'access-key-from-aws', + secretAccessKey: 'secret-access-key-from-aws', + }); + }); + }); + + it('should use awsS3.credentials if they are provided', async () => { + await jest.isolateModulesAsync(async () => { + jest.doMock('@aws-sdk/client-s3', () => ({ + ...jest.requireActual('@aws-sdk/client-s3'), + S3Client: jest.fn(), + })); + + const { S3Client: MockS3Client } = require('@aws-sdk/client-s3'); + const { AwsS3Publish: IsolatedAwsS3Publish } = require('./awsS3'); + const mockConfig = new ConfigReader({ + techdocs: { + publisher: { + type: 'awsS3', + awsS3: { + credentials: { + accessKeyId: 'accessKeyId', + secretAccessKey: 'secretAccessKey', + }, + bucketName: 'bucketName', + bucketRootPath: '/', + }, + }, + }, + integrations: { + awsS3: [ + { + accessKeyId: 'access-key-from-integrations', + secretAccessKey: 'secret-access-key-from-integrations', + }, + ], + }, + }); + + await IsolatedAwsS3Publish.fromConfig(mockConfig, logger); + expect(getCredProviderMock).toHaveBeenCalledTimes(0); + expect(MockS3Client).toHaveBeenCalledTimes(1); + await expect( + MockS3Client.mock.calls[0][0]!.credentialDefaultProvider!( + undefined!, + )(), + ).resolves.toEqual({ + accessKeyId: 'accessKeyId', + secretAccessKey: 'secretAccessKey', + }); + }); + }); + + it('should use credentials from integrations if awsS3.credentials is not provided', async () => { + await jest.isolateModulesAsync(async () => { + jest.doMock('@aws-sdk/client-s3', () => ({ + ...jest.requireActual('@aws-sdk/client-s3'), + S3Client: jest.fn(), + })); + + const { S3Client: MockS3Client } = require('@aws-sdk/client-s3'); + const { AwsS3Publish: IsolatedAwsS3Publish } = require('./awsS3'); + + const mockConfig = new ConfigReader({ + techdocs: { + publisher: { + type: 'awsS3', + awsS3: { + credentials: {}, + bucketName: 'bucketName', + bucketRootPath: '/', + }, + }, + }, + integrations: { + awsS3: [ + { + accessKeyId: 'access-key-from-integrations', + secretAccessKey: 'secret-access-key-from-integrations', + }, + ], + }, + }); + + await IsolatedAwsS3Publish.fromConfig(mockConfig, logger); + expect(getCredProviderMock).toHaveBeenCalledTimes(0); + expect(MockS3Client).toHaveBeenCalledTimes(1); + await expect( + MockS3Client.mock.calls[0][0]!.credentialDefaultProvider!( + undefined!, + )(), + ).resolves.toEqual({ + accessKeyId: 'access-key-from-integrations', + secretAccessKey: 'secret-access-key-from-integrations', + }); + }); + }); + + it('should retrieve default credentials if multiple integrations are present', async () => { const mockConfig = new ConfigReader({ techdocs: { publisher: { type: 'awsS3', awsS3: { - credentials: { - accessKeyId: 'accessKeyId', - secretAccessKey: 'secretAccessKey', - }, + credentials: {}, bucketName: 'bucketName', bucketRootPath: '/', }, }, }, + integrations: { + awsS3: [ + { + accessKeyId: 'access-key-from-integrations', + secretAccessKey: 'secret-access-key-from-integrations', + }, + { + accessKeyId: 'access-key-from-integrations-2', + secretAccessKey: 'secret-access-key-from-integrations-2', + }, + ], + }, }); await AwsS3Publish.fromConfig(mockConfig, logger); - expect(getCredProviderMock).toHaveBeenCalledTimes(0); + expect(getCredProviderMock).toHaveBeenCalledTimes(1); + }); + + it('should retrieve the target integration if multiple integrations are provided and credentials are not provided', async () => { + await jest.isolateModulesAsync(async () => { + jest.doMock('@aws-sdk/client-s3', () => ({ + ...jest.requireActual('@aws-sdk/client-s3'), + S3Client: jest.fn(), + })); + + const { S3Client: MockS3Client } = require('@aws-sdk/client-s3'); + const { AwsS3Publish: IsolatedAwsS3Publish } = require('./awsS3'); + + const mockConfig = new ConfigReader({ + techdocs: { + publisher: { + type: 'awsS3', + awsS3: { + credentials: { + accessKeyId: 'access-key-from-integrations-2', + }, + bucketName: 'bucketName', + bucketRootPath: '/', + }, + }, + }, + integrations: { + awsS3: [ + { + accessKeyId: 'access-key-from-integrations', + secretAccessKey: 'secret-access-key-from-integrations', + }, + { + accessKeyId: 'access-key-from-integrations-2', + secretAccessKey: 'secret-access-key-from-integrations-2', + }, + ], + }, + }); + + await IsolatedAwsS3Publish.fromConfig(mockConfig, logger); + expect(getCredProviderMock).toHaveBeenCalledTimes(0); + expect(MockS3Client).toHaveBeenCalledTimes(1); + await expect( + MockS3Client.mock.calls[0][0]!.credentialDefaultProvider!( + undefined!, + )(), + ).resolves.toEqual({ + accessKeyId: 'access-key-from-integrations-2', + secretAccessKey: 'secret-access-key-from-integrations-2', + }); + }); }); }); diff --git a/plugins/techdocs-node/src/stages/publish/awsS3.ts b/plugins/techdocs-node/src/stages/publish/awsS3.ts index 157c2ffdeb..ee421fd366 100644 --- a/plugins/techdocs-node/src/stages/publish/awsS3.ts +++ b/plugins/techdocs-node/src/stages/publish/awsS3.ts @@ -67,6 +67,7 @@ import { TechDocsMetadata, } from './types'; import { LoggerService } from '@backstage/backend-plugin-api'; +import { AwsS3Integration, ScmIntegrations } from '@backstage/integration'; const streamToBuffer = (stream: Readable): Promise => { return new Promise((resolve, reject) => { @@ -148,9 +149,16 @@ export class AwsS3Publish implements PublisherBase { const credentialsConfig = config.getOptionalConfig( 'techdocs.publisher.awsS3.credentials', ); + const credsManager = DefaultAwsCredentialsManager.fromConfig(config); + + const scmIntegrations = ScmIntegrations.fromConfig(config); + const awsS3Integrations = scmIntegrations.awsS3.list(); + const sdkCredentialProvider = await AwsS3Publish.buildCredentials( credsManager, + logger, + awsS3Integrations, accountId, credentialsConfig, region, @@ -227,8 +235,10 @@ export class AwsS3Publish implements PublisherBase { private static async buildCredentials( credsManager: AwsCredentialsManager, + logger: LoggerService, + awsS3Integrations: AwsS3Integration[], accountId?: string, - config?: Config, + credentialsConfig?: Config, region?: string, ): Promise { // Pull credentials for the specified account ID from the 'aws' config section @@ -237,21 +247,14 @@ export class AwsS3Publish implements PublisherBase { .sdkCredentialProvider; } - // Fall back to the default credential chain if neither account ID - // nor explicit credentials are provided - if (!config) { - return (await credsManager.getCredentialProvider()).sdkCredentialProvider; - } + const explicitCredentials = await AwsS3Publish.getExplicitCredentials({ + credsManager, + credentialsConfig, + awsS3Integrations, + logger, + }); - // Pull credentials from the techdocs config section (deprecated) - const accessKeyId = config.getOptionalString('accessKeyId'); - const secretAccessKey = config.getOptionalString('secretAccessKey'); - const explicitCredentials: AwsCredentialIdentityProvider = - accessKeyId && secretAccessKey - ? AwsS3Publish.buildStaticCredentials(accessKeyId, secretAccessKey) - : (await credsManager.getCredentialProvider()).sdkCredentialProvider; - - const roleArn = config.getOptionalString('roleArn'); + const roleArn = credentialsConfig?.getOptionalString('roleArn'); if (roleArn) { return fromTemporaryCredentials({ masterCredentials: explicitCredentials, @@ -349,6 +352,74 @@ export class AwsS3Publish implements PublisherBase { ); } + private static async getExplicitCredentials({ + credentialsConfig, + awsS3Integrations, + credsManager, + logger, + }: { + credentialsConfig?: Config; + awsS3Integrations: AwsS3Integration[]; + credsManager: AwsCredentialsManager; + logger: LoggerService; + }): Promise { + const accessKeyId = credentialsConfig?.getOptionalString('accessKeyId'); + const secretAccessKey = + credentialsConfig?.getOptionalString('secretAccessKey'); + + if (accessKeyId && secretAccessKey) { + return AwsS3Publish.buildStaticCredentials(accessKeyId, secretAccessKey); + } + + if (awsS3Integrations.length > 0) { + if (awsS3Integrations.length === 1) { + const singleAwsS3IntegrationConfig = awsS3Integrations[0].config; + + const singleAwsS3IntegrationAccessKeyId = + singleAwsS3IntegrationConfig.accessKeyId; + + const singleAwsS3IntegrationSecretAccessKey = + singleAwsS3IntegrationConfig.secretAccessKey; + + if ( + singleAwsS3IntegrationAccessKeyId && + singleAwsS3IntegrationSecretAccessKey + ) { + return AwsS3Publish.buildStaticCredentials( + singleAwsS3IntegrationAccessKeyId, + singleAwsS3IntegrationSecretAccessKey, + ); + } + } else { + if (accessKeyId) { + const targetAwsS3IntegrationConfig = awsS3Integrations.find( + c => c.config.accessKeyId === accessKeyId, + ); + + if (!targetAwsS3IntegrationConfig) { + logger.warn( + `No AWS S3 integration config under integrations.awsS3 found for access key id ${accessKeyId}.`, + ); + } + const targetAwsS3IntegrationAccessKeyId = + targetAwsS3IntegrationConfig?.config.accessKeyId; + const targetAwsS3IntegrationSecretAccessKey = + targetAwsS3IntegrationConfig?.config.secretAccessKey; + if ( + targetAwsS3IntegrationAccessKeyId && + targetAwsS3IntegrationSecretAccessKey + ) { + return AwsS3Publish.buildStaticCredentials( + targetAwsS3IntegrationAccessKeyId, + targetAwsS3IntegrationSecretAccessKey, + ); + } + } + } + } + return (await credsManager.getCredentialProvider()).sdkCredentialProvider; + } + /** * Check if the defined bucket exists. Being able to connect means the configuration is good * and the storage client will work.