paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 23 Packages Projects Releases Wiki Activity

Merge pull request #4026 from backstage/orkohunter/improve-aws-credentials-handling

Browse Source
This commit is contained in:
Himanshu Mishra
2021-01-12 15:33:25 +01:00
committed by GitHub
parent 63b92ca5f0 fda570687e
commit c3d90881ce
6 changed files with 119 additions and 156 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.changeset/stale-cougars-wink.md
+10
View File
@@ -0,0 +1,10 @@
---
'@backstage/techdocs-common': patch
'@backstage/plugin-techdocs': patch
---
AWS S3 authentication in TechDocs has been improved.
1. `techdocs.publisher.awsS3.bucketName` is now the only required config. `techdocs.publisher.awsS3.credentials` and `techdocs.publisher.awsS3.region` are optional.
2. If `techdocs.publisher.awsS3.credentials` and `techdocs.publisher.awsS3.region` are missing, the AWS environment variables `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_REGION` will be used. There are more better ways of setting up AWS authentication. Read the guide at https://backstage.io/docs/features/techdocs/using-cloud-storage
.github/styles/vocab.txt
+1
View File
@@ -87,6 +87,7 @@ Henneke
Heroku
horizontalpodautoscalers
Hostname
html
http
https
Iain
docs/features/techdocs/configuration.md
+12 -5
View File
@@ -66,15 +66,22 @@ techdocs:
# Required when techdocs.publisher.type is set to 'awsS3'. Skip otherwise.
awsS3:
# An API key is required to write to a storage bucket.
# (Required) AWS S3 Bucket Name
bucketName: 'techdocs-storage'
# (Optional) An API key is required to write to a storage bucket.
# If not set, environment variables or aws config file will be used to authenticate.
# https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/loading-node-credentials-environment.html
# https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/loading-node-credentials-shared.html
credentials:
accessKeyId:
$env: TECHDOCS_AWSS3_ACCESS_KEY_ID_CREDENTIAL
secretAccessKey:
$env: TECHDOCS_AWSS3_SECRET_ACCESS_KEY_CREDENTIAL
region:
$env: AWSS3_REGION
# AWS S3 Bucket Name
bucketName: 'techdocs-storage'
# (Optional) AWS Region of the bucket.
# If not set, AWS_REGION environment variable or aws config file will be used.
# https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/setting-region.html
region:
$env: AWS_REGION
```
docs/features/techdocs/using-cloud-storage.md
+48 -127
View File
@@ -30,7 +30,7 @@ techdocs:
type: 'googleGcs'
```
**2. GCS Bucket**
**2. Create a GCS Bucket**
Create a dedicated Google Cloud Storage bucket for TechDocs sites.
techdocs-backend will publish documentation to this bucket. TechDocs will fetch
@@ -106,10 +106,6 @@ store and read the static generated documentation files.
## Configuring AWS S3 Bucket with TechDocs
Follow the
[official AWS S3 documentation](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html)
for the latest instructions on the following steps involving AWS S3.
**1. Set `techdocs.publisher.type` config in your `app-config.yaml`**
Set `techdocs.publisher.type` to `'awsS3'`.
@@ -120,42 +116,17 @@ techdocs:
type: 'awsS3'
```
**2. AWS Policies**
**2. Create an S3 Bucket**
AWS Policies lets you **control access** to Amazon Web Services (AWS) products
and resources. Here we will use a user policy **and** a bucket policy to show
you the different possibilities you have but you can use only one.
Create a dedicated AWS S3 bucket for the storage of TechDocs sites.
[Refer to the official documentation](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html).
<img data-zoomable src="../../assets/techdocs/aws-s3.drawio.svg" alt="AWS S3" width="500" />
TechDocs will publish documentation to this bucket and will fetch files from
here to serve documentation in Backstage. Note that the bucket names are
globally unique.
This is an example of how you can manage your policies:
a. Admin user creates a **bucket policy** granting a set of permissions to our
TechDocs user.
b. Admin user attaches a **user policy** to the TechDocs user granting
additional permissions.
c. TechDocs User then tries permissions granted via both the **bucket** policy
and the **user** policy.
**2.1 Creation**
**2.1.1 Create an Admin user** (if you don't have one yet)
Create an **administrator user** account `ADMIN_USER` and grant it administrator
privileges by attaching a user policy giving the account **full access**. Note
down the Admin User credentials and IAM User Sign-In URL as you will need to use
this information in the next step.
**2.1.2 Create an AWS S3 Bucket**
Using the credentials of your Admin User `ADMIN_USER`, and the special IAM user
sign-in URL, create a dedicated **bucket** for TechDocs sites. techdocs-backend
will publish documentation to this bucket. TechDocs will fetch files from here
to serve documentation in Backstage.
Set the name of the bucket to `techdocs.publisher.awsS3.bucketName`.
Set the config `techdocs.publisher.awsS3.bucketName` in your `app-config.yaml`
to the name of the bucket you just created.
```yaml
techdocs:
@@ -165,112 +136,62 @@ techdocs:
bucketName: 'name-of-techdocs-storage-bucket'
```
**2.1.3 Create the `TechDocs` user**
**3a. (Recommended) Setup authentication the AWS way, using environment
variables**
This user will be used to interact with your bucket, it will only have
permissions to **get - put** objects.
You should follow the
[AWS security best practices guide for authentication](https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html).
In the IAM console, do the following:
If the environment variables
- Create a new user, `TechDocs`
- Note down the TechDocs User credentials
- Note down the Amazon Resource Name (ARN) for the TechDocs user. In the IAM
console, select the TechDocs user, and you can find the user ARN in the
Summary tab.
- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`
- `AWS_REGION`
**2.2 Attach policies**
are set and can be used to access the bucket you created in step 2, they will be
used by the AWS SDK v3 Node.js client for authentication.
[Refer to the official documentation.](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/loading-node-credentials-environment.html)
Remember that you can use Bucket policy **or** User policy. Just make sure that
you grant all the permissions to the TechDocs user: `s3:PutObject`,
`s3:GetObject`, `s3:ListBucket` and `s3:GetBucketLocation`.
If the environment variables are missing, the AWS SDK tries to read the
`~/.aws/credentials` file for credentials.
[Refer to the official documentation.](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/loading-node-credentials-shared.html)
**2.2.1 Create the bucket policy**
Note that the region of the bucket has to be set for the AWS SDK to work.
[See this](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/setting-region.html).
You now have to attach the following policy to your bucket in the Permission
section:
**3b. Authentication using app-config.yaml**
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "statement1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::YOUR_ACCOUNT_ID:user/TechDocs"
},
"Action": ["s3:GetBucketLocation", "s3:ListBucket"],
"Resource": ["arn:aws:s3:::name-of-techdocs-storage-bucket"]
},
{
"Sid": "statement2",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::YOUR_ACCOUNT_ID:user/TechDocs"
},
"Action": ["s3:GetObject"],
"Resource": ["arn:aws:s3:::name-of-techdocs-storage-bucket/*"]
}
]
}
```
- The first statement grants **TechDocs User** the bucket operation permissions
`s3:GetBucketLocation` and `s3:ListBucket` which are permissions required by
the console.
- The second statement grants the `s3:GetObject` permission. (**NOTE :** if you
do not use the user policy defined below you must also add the `s3:PutObject`
permission to allow the TechDocs user to add objects.)
**2.2.2 Create the user policy**
Create an inline policy for the TechDocs user by using the following policy:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PermissionForObjectOperations",
"Effect": "Allow",
"Action": ["s3:PutObject"],
"Resource": ["arn:aws:s3:::name-of-techdocs-storage-bucket/*"]
}
]
}
```
See more details in the section
[Working with Inline Policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage.html).
Now you need to fill in the environment variables with the `TechDocs` User
credentials. You can also specify a region if you want to accesses the resources
in a specific region. Otherwise no region will be selected by default.
```properties
TECHDOCS_AWSS3_ACCESS_KEY_ID_CREDENTIAL="TECHDOCS_ACCESS_KEY_ID"
TECHDOCS_AWSS3_SECRET_ACCESS_KEY_CREDENTIAL="TECHDOCS_SECRET_ACCESS_KEY"
AWSS3_REGION="" // Optional
```
Make it available in your Backstage server and/or your local development server
and set it in the app config techdocs.publisher.awsS3.
AWS credentials and region can be provided to the AWS SDK via `app-config.yaml`.
If the configs below are present, they will be used over existing `AWS_*`
environment variables and the `~/.aws/credentials` config file.
```yaml
techdocs:
publisher:
type: 'awsS3'
awsS3:
bucketName: 'name-of-techdocs-storage-bucket'
region:
$env: AWS_REGION
credentials:
accessKeyId:
$env: TECHDOCS_AWSS3_ACCESS_KEY_ID_CREDENTIAL
$env: AWS_ACCESS_KEY_ID
secretAccessKey:
$env: TECHDOCS_AWSS3_SECRET_ACCESS_KEY_CREDENTIAL
region:
$env: AWSS3_REGION
$env: AWS_SECRET_ACCESS_KEY
```
**3. That's it!**
Refer to the
[official AWS documentation for obtaining the credentials](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/getting-your-credentials.html).
Your Backstage app is now ready to use AWS S3 for TechDocs, to store the static
generated documentation files.
Note: If you are using Amazon EC2 instance to deploy Backstage, you do not need
to obtain the access keys separately. They can be made available in the
environment automatically by defining appropriate IAM role with access to the
bucket. Read more
[here](https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html#use-roles).
**4. That's it!**
Your Backstage app is now ready to use AWS S3 for TechDocs, to store and read
the static generated documentation files. When you start the backend of the app,
you should be able to see
`techdocs info Successfully connected to the AWS S3 bucket` in the logs.
packages/techdocs-common/src/stages/publish/awsS3.ts
+37 -18
View File
@@ -39,30 +39,48 @@ const streamToString = (stream: Readable): Promise<string> => {
export class AwsS3Publish implements PublisherBase {
static fromConfig(config: Config, logger: Logger): PublisherBase {
let region = null;
let accessKeyId = null;
let secretAccessKey = null;
let bucketName = '';
try {
accessKeyId = config.getString(
'techdocs.publisher.awsS3.credentials.accessKeyId',
);
secretAccessKey = config.getString(
'techdocs.publisher.awsS3.credentials.secretAccessKey',
);
region = config.getOptionalString('techdocs.publisher.awsS3.region');
bucketName = config.getString('techdocs.publisher.awsS3.bucketName');
} catch (error) {
throw new Error(
"Since techdocs.publisher.type is set to 'awsS3' in your app config, " +
'credentials and bucketName are required in techdocs.publisher.awsS3 ' +
'required to authenticate with AWS S3.',
'techdocs.publisher.awsS3.bucketName is required.',
);
}
// Credentials is an optional config. If missing, default AWS environment variables
// or AWS shared credentials file at ~/.aws/credentials will be used to authenticate
// https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/loading-node-credentials-environment.html
// https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/loading-node-credentials-shared.html
const credentials = config.getOptionalConfig(
'techdocs.publisher.awsS3.credentials',
);
let accessKeyId = undefined;
let secretAccessKey = undefined;
if (credentials) {
accessKeyId = credentials.getOptionalString('accessKeyId');
secretAccessKey = credentials.getOptionalString('secretAccessKey');
}
// AWS Region is an optional config. If missing, default AWS env variable AWS_REGION
// or AWS shared credentials file at ~/.aws/credentials will be used. Any way, AWS SDK v3 client needs
// to have the AWS Region information for it to work.
// https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/setting-region.html
const region = config.getOptionalString('techdocs.publisher.awsS3.region');
const storageClient = new S3({
credentials: { accessKeyId, secretAccessKey },
...(region && { region }),
...(credentials &&
accessKeyId &&
secretAccessKey && {
credentials: {
accessKeyId,
secretAccessKey,
},
}),
...(region && {
region,
}),
});
// Check if the defined bucket exists. Being able to connect means the configuration is good
@@ -75,11 +93,12 @@ export class AwsS3Publish implements PublisherBase {
if (err) {
logger.error(
`Could not retrieve metadata about the AWS S3 bucket ${bucketName}. ` +
'Make sure the AWS project and the bucket exists and the access key located at the path ' +
"techdocs.publisher.awsS3.credentials defined in app config has the role 'Storage Object Creator'. " +
'Refer to https://backstage.io/docs/features/techdocs/using-cloud-storage',
'Make sure the bucket exists. Also make sure that authentication is setup either by ' +
'explicitly defining credentials and region in techdocs.publisher.awsS3 in app config or ' +
'by using environment variables. Refer to https://backstage.io/docs/features/techdocs/using-cloud-storage',
);
throw new Error(`from AWS client library: ${err.message}`);
logger.error(`from AWS client library: ${err.message}`);
throw new Error();
} else {
logger.info(
`Successfully connected to the AWS S3 bucket ${bucketName}.`,
plugins/techdocs/config.d.ts
Vendored
+11 -6
View File
@@ -78,10 +78,13 @@ export interface Config {
*/
awsS3?: {
/**
* Credentials used to access a storage bucket
* (Optional) Credentials used to access a storage bucket.
* If not set, environment variables or aws config file will be used to authenticate.
* https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/loading-node-credentials-environment.html
* https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/loading-node-credentials-shared.html
* @visibility secret
*/
credentials: {
credentials?: {
/**
* User access key id
* attr: 'accessKeyId' - accepts a string value
@@ -96,13 +99,15 @@ export interface Config {
secretAccessKey: string;
};
/**
* Cloud Storage Bucket Name
* (Required) Cloud Storage Bucket Name
* attr: 'bucketName' - accepts a string value
* @visibility secret
* @visibility backend
*/
bucketName: string;
/**
* AWS Region
* (Optional) AWS Region.
* If not set, AWS_REGION environment variable or aws config file will be used.
* https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/setting-region.html
* attr: 'region' - accepts a string value
* @visibility secret
*/
@@ -125,7 +130,7 @@ export interface Config {
/**
* (Required) Cloud Storage Bucket Name
* attr: 'bucketName' - accepts a string value
* @visibility secret
* @visibility backend
*/
bucketName: string;
/**