paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 40 Packages Projects Releases Wiki Activity

Google Cloud Storage working with TechDocs (publish and fetch)

Browse Source 
Plus
1. Introduce ncessary configs to connect with storage
2. Introduce config to prefer CI build of docs, or techdocs-backend to build and publish them.
3. Write guide how to use Google Cloud Storage with TechDocs
4. Add a TechDocs Configuration reference page in docucmentation
This commit is contained in:
Himanshu Mishra
2020-12-03 22:40:15 +01:00
parent e09cff1682
commit 084670b37a
20 changed files with 971 additions and 163 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app-config.yaml
+5 -6
View File
@@ -55,16 +55,15 @@ proxy:
organization:
name: My Company
# Reference documentation http://backstage.io/docs/features/techdocs/configuration
techdocs:
storageUrl: http://localhost:7000/api/techdocs/static/docs
requestUrl: http://localhost:7000/api/techdocs
storageUrl: http://localhost:7000/api/techdocs/static/docs
docsBuilder: 'local' # Alternatives - 'ci'
generators:
techdocs: 'docker'
techdocs: 'docker' # Alternatives - 'local'
publisher:
# Ref: <Link to TechDocs publisher reference page>
type: 'local' # other options - google_gcs, aws_s3, azure_storage, etc.
# if type set to google_gcs
# google_token: 'abc' # with write access
type: 'local' # Alternatives - 'google_gcs'. Read documentation for using alternatives.
sentry:
organization: my-company
docs/features/techdocs/configuration.md
+71
View File
@@ -0,0 +1,71 @@
---
id: configuration
title: TechDocs Configuration Options
description: Reference documentation for configuring TechDocs using app-config.yaml
---
Using the `app-config.yaml` in the Backstage app, you can configure TechDocs using several options.
This page serves as a reference to all the available configuration options for TechDocs.
```yaml
# File: app-config.yaml
techdocs:
# TechDocs makes API calls to techdocs-backend using this URL. e.g. get docs of an entity, get metadata, etc.
requestUrl: http://localhost:7000/api/techdocs
# Just another route in techdocs-backend where TechDocs requests the static files from. This URL uses an HTTP middleware
# to serve files from either a local directory or an External storage provider.
storageUrl: http://localhost:7000/api/techdocs/static/docs
# generators.techdocs can have two values: 'docker' or 'local'. This is to determine how to run the generator - whether to
# spin up the techdocs-container docker image or to run mkdocs locally (assuming all the dependencies are taken care of).
# You want to change this to 'local' if you are running Backstage using your own customer Docker setup and want to avoid running
# into Docker in Docker situation. Read more here
# https://backstage.io/docs/features/techdocs/getting-started#disable-docker-in-docker-situation-optional
generators:
techdocs: 'docker'
# techdocs.builder can be either 'local' or 'ci.
# If builder is set to 'local' and you open a TechDocs page, techdocs-backend will try to build the docs, publish to storage
# and show the generated docs afterwords. This is the "Basic" setup of the TechDocs Architecture.
# If builder is set to 'ci', techdocs-backend will only fetch the docs and will NOT try to build and publish. In this case of 'ci',
# we assume that docs are being built in the CI/CD pipeline of the repository. This is the "Recommended" setup of the architecture.
# Read more here https://backstage.io/docs/features/techdocs/architecture
builder: 'local'
# techdocs.publisher is used to configure the Storage option, whether you want to use the local filesystem to store generated docs
# or you want to use External storage providers like Google Cloud Storage, AWS S3, etc.
publisher:
# techdocs.publisher.type can be - 'local' or 'google_gcs' (aws_s3, azure_storage, etc. to be available as well).
# When set to 'local', techdocs-backend will create a 'static' directory at its root to store generated documentation files.
# When set to 'google_gcs', techdocs-backend will use a Google Cloud Storage Bucket to store generated documentation files.
type: 'local'
# Required when techdocs.publisher.type is set to 'google_gcs'. Skip otherwise.
google:
# An API key is required to write to a storage bucket.
pathToKey: '/path/to/google_application_credentials.json',
# Your GCP Project ID where the Cloud Storage Bucket is hosted.
projectId: 'gcp-project-id'
# Cloud Storage Bucket Name
bucketName: 'techdocs-storage',
```
docs/features/techdocs/using-cloud-storage.md
+80
View File
@@ -0,0 +1,80 @@
---
id: using-cloud-storage
title: Using Cloud Storage for TechDocs generated files
description: Using Cloud Storage for TechDocs generated files
---
In the [TechDocs architecture](./architecture.md) you have the option to choose where you want to store
the Generated static files which TechDocs uses to render documentation. In both the "Basic" and "Recommended"
setup, you can add cloud storage providers like Google GCS, Amazon AWS S3, etc. By default, TechDocs
uses the local filesystem of the `techdocs-backend` plugin in the "Basic" setup. And in the recommended setup,
having one of the cloud storage is a prerequisite. Read more on the TechDocs Architecture documentation page.
On this page you you can read how to enable them.
## Configuring Google GCS Bucket with TechDocs
Follow the [official Google Cloud documentation](https://googleapis.dev/nodejs/storage/latest/index.html#quickstart) for the latest instructions
on the following steps involving GCP.
**1. Set `techdocs.publisher.type` config in your `app-config.yaml`**
Set `techdocs.publisher.type` to `'google_gcs'`.
```yaml
techdocs:
publisher:
type: 'google_gcs'
```
**2. GCP (Google Cloud Platform) Project**
Create or choose a dedicated GCP project. Set `techdocs.publisher.google.projectId` to the project ID.
```yaml
techdocs:
publisher:
type: 'google_gcs'
google:
projectId: 'gcp-project-id
```
**3. Service account API key**
Create a new Service Account and a key associated with it. In roles of the service account, use "Storage Admin".
If you want to create a custom role, make sure to include both `get` and `create` permissions for both "Objects" and "Buckets". See https://cloud.google.com/storage/docs/access-control/iam-permissions
A service account can have many keys. Open your newly created account's page (in IAM & Admin console), and create a new key. Use JSON format for the key.
A `<GCP-PROJECT-ID-random-uid>.json` file will be downloaded. This is the secret key TechDocs will use to make API calls. Make it available in your Backstage server and/or your local development server and set it in the app config `techdocs.publisher.google.pathToKey`.
```yaml
techdocs:
publisher:
type: 'google_gcs'
google:
projectId: 'gcp-project-id'
pathToKey: '/path/to/google_application_credentials.json'
```
**4. GCS Bucket**
Create a dedicated bucket for TechDocs sites. techdocs-backend will publish documentation to this bucket.
TechDocs will fetch files from here to serve documentation in Backstage.
Set the name of the bucket to `techdocs.publisher
```yaml
techdocs:
publisher:
type: 'google_gcs'
google:
projectId: 'gcp-project-id'
pathToKey: '/path/to/google_application_credentials.json'
bucketName: 'name-of-techdocs-storage-bucket'
```
**5. That's it!**
Your Backstage app is now ready to use Google Cloud Storage for TechDocs, to store the static generated documentation files.
microsite/sidebars.json
+2
View File
@@ -74,6 +74,8 @@
"features/techdocs/concepts",
"features/techdocs/architecture",
"features/techdocs/creating-and-publishing",
"features/techdocs/configuration",
"features/techdocs/using-cloud-storage",
"features/techdocs/troubleshooting",
"features/techdocs/faqs"
]
mkdocs.yml
+2
View File
@@ -54,6 +54,8 @@ nav:
- Concepts: 'features/techdocs/concepts.md'
- TechDocs Architecture: 'features/techdocs/architecture.md'
- Creating and Publishing Documentation: 'features/techdocs/creating-and-publishing.md'
- Configuration: 'features/techdocs/configuration.md'
- Using Cloud Storage: 'features/techdocs/using-cloud-storage.md'
- Troubleshooting: 'features/techdocs/troubleshooting.md'
- FAQ: 'features/techdocs/FAQ.md'
- Plugins:
packages/techdocs-common/package.json
+3 -1
View File
@@ -39,14 +39,16 @@
"@backstage/backend-common": "^0.3.2",
"@backstage/catalog-model": "^0.3.1",
"@backstage/config": "^0.1.1",
"@google-cloud/storage": "^5.6.0",
"@types/dockerode": "^2.5.34",
"@types/express-serve-static-core": "^4.17.14",
"@types/klaw": "^3.0.1",
"cross-fetch": "^3.0.6",
"dockerode": "^3.2.1",
"express": "^4.17.1",
"fs-extra": "^9.0.1",
"git-url-parse": "^11.4.0",
"js-yaml": "^3.14.0",
"klaw": "^3.0.0",
"mock-fs": "^4.13.0",
"nodegit": "^0.27.0",
"winston": "^3.2.1"
packages/techdocs-common/src/stages/publish/googleStorage.ts
+222
View File
@@ -0,0 +1,222 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import path from 'path';
import express from 'express';
import walk from 'klaw';
import { Storage, UploadResponse } from '@google-cloud/storage';
import { Logger } from 'winston';
import { Entity, EntityName } from '@backstage/catalog-model';
import { Config } from '@backstage/config';
import { getHeadersForFileExtension, supportedFileType } from './helpers';
import { PublisherBase, PublisherBaseParams } from './types';
export class GoogleGCSPublish implements PublisherBase {
static fromConfig(config: Config, logger: Logger): PublisherBase {
let pathToKey = '';
let projectId = '';
let bucketName = '';
try {
pathToKey = config.getString('techdocs.publisher.google.pathToKey');
projectId = config.getString('techdocs.publisher.google.projectId');
bucketName = config.getString('techdocs.publisher.google.bucketName');
} catch (error) {
throw new Error(
"Since techdocs.publisher.type is set to 'google_gcs' in your app config, " +
'pathToKey, projectId and bucketName are required in techdocs.publisher.google ' +
'required to authenticate with Google Cloud Storage.',
);
}
const storageClient = new Storage({
projectId: projectId,
keyFilename: pathToKey,
});
// Check if the defined bucket exists. Being able to connect means the configuration is good
// and the storage client will work.
storageClient
.bucket(bucketName)
.getMetadata()
.then(() => {
logger.info(
`Successfully connected to the GCS bucket ${bucketName} in the GCP project ${projectId}.`,
);
})
.catch(reason => {
logger.error(
`Could not retrieve metadata about the GCS bucket ${bucketName} in the GCP project ${projectId}. ` +
'Make sure the GCP project and the bucket exists and the access key located at the path ' +
"techdocs.publisher.google.pathToKey defined in app config has the role 'Storage Object Creator'. " +
'Refer to https://backstage.io/docs/features/techdocs/using-cloud-storage',
);
throw new Error(`from GCS client library: ${reason.message}`);
});
return new GoogleGCSPublish(storageClient, bucketName, logger);
}
constructor(
private readonly storageClient: Storage,
private readonly bucketName: string,
private readonly logger: Logger,
) {
this.storageClient = storageClient;
this.bucketName = bucketName;
this.logger = logger;
}
/**
* Upload all the files from the generated `directory` to the GCS bucket.
* Directory structure used in the bucket is - entityNamespace/entityKind/entityName/index.html
*/
publish({ entity, directory }: PublisherBaseParams): Promise<{}> {
return new Promise((resolve, reject) => {
// Path of all files to upload, relative to the root of the source directory
// e.g. ['index.html', 'sub-page/index.html', 'assets/images/favicon.png']
const allFilesToUpload: Array<string> = [];
// Iterate on all the files in the directory and its sub-directories
walk(directory)
.on('data', (item: walk.Item) => {
// GCS manages creation of parent directories if they do not exist.
// So collecting path of only the files is good enough.
if (item.stats.isFile()) {
// Remove the absolute path prefix of the source directory
const relativeFilePath = item.path.replace(`${directory}/`, '');
allFilesToUpload.push(relativeFilePath);
}
})
.on('error', (err: Error, item: walk.Item) => {
const errorMessage = `Unable to read file at ${item.path}. Error ${err.message}`;
this.logger.error(errorMessage);
reject(errorMessage);
})
.on('end', () => {
// 'end' event happens when all the files have been read.
const entityRootDir = `${entity.metadata.namespace}/${entity.kind}/${entity.metadata.name}`;
allFilesToUpload.forEach(filePath => {
const source = path.join(directory, filePath); // Local file absolutely path
const destination = `${entityRootDir}/${filePath}`; // GCS Bucket file relative path
this.storageClient
.bucket(this.bucketName)
.upload(source, { destination })
.then(
(uploadResp: UploadResponse) => ({
fileName: destination,
status: uploadResp[0],
}),
(err: Error) => {
const errorMessage = `Unable to upload file ${destination} to GCS. Error ${err.message}`;
this.logger.error(errorMessage);
reject(errorMessage);
},
);
});
this.logger.info(
`Successfully uploaded all the generated files for Entity ${entityRootDir}. Total number of files: ${allFilesToUpload.length}`,
);
resolve({});
});
});
}
fetchTechDocsMetadata(entityName: EntityName): Promise<string> {
return new Promise((resolve, reject) => {
const entityRootDir = `${entityName.namespace}/${entityName.kind}/${entityName.name}`;
const fileStreamChunks: Array<any> = [];
this.storageClient
.bucket(this.bucketName)
.file(`${entityRootDir}/techdocs_metadata.json`)
.createReadStream()
.on('error', err => {
this.logger.error(err.message);
reject(err.message);
})
.on('data', chunk => {
fileStreamChunks.push(chunk);
})
.on('end', () => {
const techdocsMetadataJson = Buffer.concat(
fileStreamChunks,
).toString();
resolve(techdocsMetadataJson);
});
});
}
/**
* Express route middleware to serve static files on a route in techdocs-backend.
*/
docsRouter(): express.Handler {
return (req, res) => {
// Trim the leading forward slash
// filePath example - /default/Component/documented-component/index.html
const filePath = req.path.replace(/^\//, '');
// Files with different extensions (CSS, HTML) need to be served with different headers
const fileExtension = filePath.split('.')[filePath.split('.').length - 1];
const responseHeaders = getHeadersForFileExtension(
fileExtension as supportedFileType,
);
const fileStreamChunks: Array<any> = [];
this.storageClient
.bucket(this.bucketName)
.file(filePath)
.createReadStream()
.on('error', err => {
this.logger.error(err.message);
res.send(err.message);
})
.on('data', chunk => {
fileStreamChunks.push(chunk);
})
.on('end', () => {
const fileContent = Buffer.concat(fileStreamChunks).toString();
// Inject response headers
for (const [headerKey, headerValue] of Object.entries(
responseHeaders,
)) {
res.setHeader(headerKey, headerValue);
}
res.send(fileContent);
});
};
}
/**
* A helper function which checks if index.html of an Entity's docs site is available. This
* can be used to verify if there are any pre-generated docs available to serve.
*/
async hasDocsBeenGenerated(entity: Entity): Promise<boolean> {
return new Promise(resolve => {
const entityRootDir = `${entity.metadata.namespace}/${entity.kind}/${entity.metadata.name}`;
this.storageClient
.bucket(this.bucketName)
.file(`${entityRootDir}/index.html`)
.createReadStream()
.on('error', () => {
resolve(false);
})
.on('data', () => {
resolve(true);
});
});
}
}
packages/techdocs-common/src/stages/publish/helpers.ts
+46
View File
@@ -0,0 +1,46 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
export type supportedFileType = 'html' | 'css';
export type responseHeadersType = {
'Content-Type': string;
};
export const getHeadersForFileExtension = (
fileType: supportedFileType,
): responseHeadersType => {
const headersCommon = {
'Content-Type': 'text/plain',
};
const headersHTML = {
...headersCommon,
'Content-Type': 'text/html; charset=UTF-8',
};
const headersCSS = {
...headersCommon,
'Content-Type': 'text/css; charset=UTF-8',
};
switch (fileType) {
case 'html':
return headersHTML;
case 'css':
return headersCSS;
default:
return headersCommon;
}
};
packages/techdocs-common/src/stages/publish/local.ts
+27 -11
View File
@@ -13,12 +13,11 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import express from 'express';
import * as expressServeStaticCore from 'express-serve-static-core'; // Type from express library
import fetch from 'cross-fetch';
import express from 'express';
import fs from 'fs-extra';
import { Logger } from 'winston';
import { EntityName } from '@backstage/catalog-model';
import { Entity, EntityName } from '@backstage/catalog-model';
import {
resolvePackagePath,
PluginEndpointDiscovery,
@@ -92,9 +91,7 @@ export class LocalPublish implements PublisherBase {
});
}
fetchTechDocsMetadata(
entityName: EntityName,
): Promise<{ techdocsMetadataJson: string }> {
fetchTechDocsMetadata(entityName: EntityName): Promise<string> {
return new Promise((resolve, reject) => {
this.discovery.getBaseUrl('techdocs').then(techdocsApiUrl => {
const storageUrl = new URL(
@@ -102,8 +99,8 @@ export class LocalPublish implements PublisherBase {
techdocsApiUrl,
).toString();
const path = `${entityName.kind}/${entityName.namespace}/${entityName.name}`;
const metadataURL = `${storageUrl}/${path}/techdocs_metadata.json`;
const entityRootDir = `${entityName.namespace}/${entityName.kind}/${entityName.name}`;
const metadataURL = `${storageUrl}/${entityRootDir}/techdocs_metadata.json`;
fetch(metadataURL)
.then(response =>
response
@@ -111,18 +108,37 @@ export class LocalPublish implements PublisherBase {
.then(techdocsMetadataJson => resolve(techdocsMetadataJson))
.catch(err => {
reject(
`Unable to parse metadata JSON for ${path}. Error: ${err}`,
`Unable to parse metadata JSON for ${entityRootDir}. Error: ${err}`,
);
}),
)
.catch(err => {
reject(`Unable to fetch metadata for ${path}. Error ${err}`);
reject(
`Unable to fetch metadata for ${entityRootDir}. Error ${err}`,
);
});
});
});
}
docsRouter(): expressServeStaticCore.Handler {
docsRouter(): express.Handler {
return express.static(staticDocsDir);
}
async hasDocsBeenGenerated(entity: Entity): Promise<boolean> {
return new Promise(resolve => {
this.discovery.getBaseUrl('techdocs').then(techdocsApiUrl => {
const storageUrl = new URL(
new URL(this.config.getString('techdocs.storageUrl')).pathname,
techdocsApiUrl,
).toString();
const entityRootDir = `${entity.metadata.namespace}/${entity.kind}/${entity.metadata.name}`;
const indexHtmlUrl = `${storageUrl}/${entityRootDir}/index.html`;
fetch(indexHtmlUrl)
.then(() => resolve(true))
.catch(() => resolve(false));
});
});
}
}
packages/techdocs-common/src/stages/publish/publish.ts
+2 -1
View File
@@ -19,6 +19,7 @@ import { PluginEndpointDiscovery } from '@backstage/backend-common';
import { PublisherType, PublisherBase } from './types';
import { LocalPublish } from './local';
import { GoogleGCSPublish } from './googleStorage';
/**
* Factory class to create a TechDocs publisher based on defined publisher type in app config.
@@ -37,7 +38,7 @@ export class Publisher {
switch (publisherType) {
case 'google_gcs':
logger.info('Creating Google Storage Bucket publisher for TechDocs');
return new LocalPublish(config, logger, discovery);
return GoogleGCSPublish.fromConfig(config, logger);
case 'local':
logger.info('Creating Local publisher for TechDocs');
return new LocalPublish(config, logger, discovery);
packages/techdocs-common/src/stages/publish/types.ts
+8 -7
View File
@@ -14,9 +14,7 @@
* limitations under the License.
*/
import { Entity, EntityName } from '@backstage/catalog-model';
// import serveStatic from 'serve-static';
// import express from 'express';
import * as express from 'express-serve-static-core';
import express from 'express';
/**
* Key for all the different types of TechDocs publishers that are supported.
@@ -49,12 +47,15 @@ export interface PublisherBase {
* Retrieve TechDocs Metadata about a site e.g. name, contributors, last updated, etc.
* This API uses the techdocs_metadata.json file that co-exists along with the generated docs.
*/
fetchTechDocsMetadata(
entityName: EntityName,
): Promise<{ techdocsMetadataJson: string }>;
fetchTechDocsMetadata(entityName: EntityName): Promise<string>;
/**
*
* Route middleware to serve static documentation files for an entity.
*/
docsRouter(): express.Handler;
/**
* Check if the index.html is present for the Entity at the Storage location.
*/
hasDocsBeenGenerated(entityName: Entity): Promise<boolean>;
}
plugins/techdocs-backend/src/storage/BuildMetadataStorage.test.ts → plugins/techdocs-backend/src/DocsBuilder/BuildMetadataStorage.test.ts
View File
plugins/techdocs-backend/src/storage/BuildMetadataStorage.ts → plugins/techdocs-backend/src/DocsBuilder/BuildMetadataStorage.ts
View File
plugins/techdocs-backend/src/DocsBuilder/builder.ts
+129
View File
@@ -0,0 +1,129 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import Docker from 'dockerode';
import { Logger } from 'winston';
import { Entity } from '@backstage/catalog-model';
import {
PreparerBuilder,
PublisherBase,
GeneratorBuilder,
PreparerBase,
GeneratorBase,
getLocationForEntity,
getLastCommitTimestamp,
} from '@backstage/techdocs-common';
import { BuildMetadataStorage } from '.';
const getEntityId = (entity: Entity) => {
return `${entity.kind}:${entity.metadata.namespace ?? ''}:${
entity.metadata.name
}`;
};
type DocsBuilderArguments = {
preparers: PreparerBuilder;
generators: GeneratorBuilder;
publisher: PublisherBase;
entity: Entity;
logger: Logger;
dockerClient: Docker;
};
export class DocsBuilder {
private preparer: PreparerBase;
private generator: GeneratorBase;
private publisher: PublisherBase;
private entity: Entity;
private logger: Logger;
private dockerClient: Docker;
constructor({
preparers,
generators,
publisher,
entity,
logger,
dockerClient,
}: DocsBuilderArguments) {
this.preparer = preparers.get(entity);
this.generator = generators.get(entity);
this.publisher = publisher;
this.entity = entity;
this.logger = logger;
this.dockerClient = dockerClient;
}
public async build() {
this.logger.info(`Running preparer on entity ${getEntityId(this.entity)}`);
const preparedDir = await this.preparer.prepare(this.entity);
const parsedLocationAnnotation = getLocationForEntity(this.entity);
this.logger.info(`Running generator on entity ${getEntityId(this.entity)}`);
const { resultDir } = await this.generator.run({
directory: preparedDir,
dockerClient: this.dockerClient,
parsedLocationAnnotation,
});
this.logger.info(`Running publisher on entity ${getEntityId(this.entity)}`);
await this.publisher.publish({
entity: this.entity,
directory: resultDir,
});
if (!this.entity.metadata.uid) {
throw new Error(
'Trying to build documentation for entity not in service catalog',
);
}
new BuildMetadataStorage(this.entity.metadata.uid).storeBuildTimestamp();
}
public async docsUpToDate() {
if (!this.entity.metadata.uid) {
throw new Error(
'Trying to build documentation for entity not in service catalog',
);
}
const buildMetadataStorage = new BuildMetadataStorage(
this.entity.metadata.uid,
);
const { type, target } = getLocationForEntity(this.entity);
// Unless docs are stored locally
const nonAgeCheckTypes = ['dir', 'file', 'url'];
if (!nonAgeCheckTypes.includes(type)) {
const lastCommit = await getLastCommitTimestamp(target, this.logger);
const storageTimeStamp = buildMetadataStorage.getTimestamp();
// Check if documentation source is newer than what we have
if (storageTimeStamp && storageTimeStamp >= lastCommit) {
this.logger.debug(
`Docs for entity ${getEntityId(this.entity)} is up to date.`,
);
return true;
}
}
this.logger.debug(
`Docs for entity ${getEntityId(this.entity)} was outdated.`,
);
return false;
}
}
plugins/techdocs-backend/src/storage/index.ts → plugins/techdocs-backend/src/DocsBuilder/index.ts
+1
View File
@@ -14,3 +14,4 @@
* limitations under the License.
*/
export * from './BuildMetadataStorage';
export * from './builder';
plugins/techdocs-backend/src/index.ts
+1
View File
@@ -15,4 +15,5 @@
*/
export * from './service/router';
// TODO: Do named exports here e.g. publishers, generators.
export * from '@backstage/techdocs-common';
plugins/techdocs-backend/src/service/helpers.test.ts
+26
View File
@@ -0,0 +1,26 @@
/*
* Copyright 2020 Spotify AB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import { getEntityNameFromUrlPath } from './helpers';
describe('getEntityNameFromUrlPath', () => {
it('should parse correctly', () => {
const path = 'default/Component/documented-component';
const parsedEntity = getEntityNameFromUrlPath(path);
expect(parsedEntity).toHaveProperty('namespace', 'default');
expect(parsedEntity).toHaveProperty('kind', 'Component');
expect(parsedEntity).toHaveProperty('name', 'documented-component');
});
});
plugins/techdocs-backend/src/service/helpers.ts
+3 -117
View File
@@ -13,132 +13,18 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import Docker from 'dockerode';
import { Logger } from 'winston';
import { Entity, EntityName } from '@backstage/catalog-model';
import {
PreparerBuilder,
PublisherBase,
GeneratorBuilder,
PreparerBase,
GeneratorBase,
getLocationForEntity,
getLastCommitTimestamp,
} from '@backstage/techdocs-common';
import { BuildMetadataStorage } from '../storage';
const getEntityId = (entity: Entity) => {
return `${entity.kind}:${entity.metadata.namespace ?? ''}:${
entity.metadata.name
}`;
};
type DocsBuilderArguments = {
preparers: PreparerBuilder;
generators: GeneratorBuilder;
publisher: PublisherBase;
entity: Entity;
logger: Logger;
dockerClient: Docker;
};
export class DocsBuilder {
private preparer: PreparerBase;
private generator: GeneratorBase;
private publisher: PublisherBase;
private entity: Entity;
private logger: Logger;
private dockerClient: Docker;
constructor({
preparers,
generators,
publisher,
entity,
logger,
dockerClient,
}: DocsBuilderArguments) {
this.preparer = preparers.get(entity);
this.generator = generators.get(entity);
this.publisher = publisher;
this.entity = entity;
this.logger = logger;
this.dockerClient = dockerClient;
}
public async build() {
this.logger.info(`Running preparer on entity ${getEntityId(this.entity)}`);
const preparedDir = await this.preparer.prepare(this.entity);
const parsedLocationAnnotation = getLocationForEntity(this.entity);
this.logger.info(`Running generator on entity ${getEntityId(this.entity)}`);
const { resultDir } = await this.generator.run({
directory: preparedDir,
dockerClient: this.dockerClient,
parsedLocationAnnotation,
});
this.logger.info(`Running publisher on entity ${getEntityId(this.entity)}`);
await this.publisher.publish({
entity: this.entity,
directory: resultDir,
});
if (!this.entity.metadata.uid) {
throw new Error(
'Trying to build documentation for entity not in service catalog',
);
}
new BuildMetadataStorage(this.entity.metadata.uid).storeBuildTimestamp();
}
public async docsUpToDate() {
if (!this.entity.metadata.uid) {
throw new Error(
'Trying to build documentation for entity not in service catalog',
);
}
const buildMetadataStorage = new BuildMetadataStorage(
this.entity.metadata.uid,
);
const { type, target } = getLocationForEntity(this.entity);
// Unless docs are stored locally
const nonAgeCheckTypes = ['dir', 'file', 'url'];
if (!nonAgeCheckTypes.includes(type)) {
const lastCommit = await getLastCommitTimestamp(target, this.logger);
const storageTimeStamp = buildMetadataStorage.getTimestamp();
// Check if documentation source is newer than what we have
if (storageTimeStamp && storageTimeStamp >= lastCommit) {
this.logger.debug(
`Docs for entity ${getEntityId(this.entity)} is up to date.`,
);
return true;
}
}
this.logger.debug(
`Docs for entity ${getEntityId(this.entity)} was outdated.`,
);
return false;
}
}
import { EntityName } from '@backstage/catalog-model';
/**
* Using the path of the TechDocs page URL, return a structured EntityName type object with namespace,
* kind and name of the Entity.
* @param {string} path Example: default/Component/documented-component
*/
export const getEntityNameFromUrlPath = (path: string): EntityName => {
const [kind, namespace, name] = path.split('/');
const [namespace, kind, name] = path.split('/');
return {
kind,
namespace,
kind,
name,
};
};
plugins/techdocs-backend/src/service/router.ts
+46 -13
View File
@@ -28,7 +28,8 @@ import {
} from '@backstage/techdocs-common';
import { PluginEndpointDiscovery } from '@backstage/backend-common';
import { Entity } from '@backstage/catalog-model';
import { DocsBuilder, getEntityNameFromUrlPath } from './helpers';
import { getEntityNameFromUrlPath } from './helpers';
import { DocsBuilder } from '../DocsBuilder';
type RouterOptions = {
preparers: PreparerBuilder;
@@ -92,9 +93,8 @@ export async function createRouter({
});
router.get('/docs/:namespace/:kind/:name/*', async (req, res) => {
const storageUrl = config.getString('techdocs.storageUrl');
const { kind, namespace, name } = req.params;
const storageUrl = config.getString('techdocs.storageUrl');
const catalogUrl = await discovery.getBaseUrl('catalog');
const triple = [kind, namespace, name].map(encodeURIComponent).join('/');
@@ -109,22 +109,55 @@ export async function createRouter({
const entity: Entity = await catalogRes.json();
const docsBuilder = new DocsBuilder({
preparers,
generators,
publisher,
dockerClient,
logger,
entity,
});
let publisherType = '';
try {
publisherType = config.getString('techdocs.publisher.type');
} catch (err) {
throw new Error(
'Unable to get techdocs.publisher.type in your app config. Set it to either ' +
"'local', 'google_gcs' or other support storage providers. Read more here " +
'https://backstage.io/docs/features/techdocs/architecture',
);
}
if (!(await docsBuilder.docsUpToDate())) {
await docsBuilder.build();
// techdocs-backend will only try to build documentation for an entity if techdocs.docsBuilder is set to 'local'
// If set to 'ci', it will only try to fetch and assume that that CI/CD pipeline of the repository hosting the
// entity's documentation is responsible for building and publishing documentation to the storage provider.
const shouldBuildDocs =
config.getString('techdocs.docsBuilder') === 'local';
if (shouldBuildDocs) {
const docsBuilder = new DocsBuilder({
preparers,
generators,
publisher,
dockerClient,
logger,
entity,
});
if (publisherType === 'local') {
if (!(await docsBuilder.docsUpToDate())) {
await docsBuilder.build();
}
} else if (publisherType === 'google_gcs') {
if (!(await publisher.hasDocsBeenGenerated(entity))) {
logger.info(
'Did not find generated docs files for the entity. Building docs.',
);
await docsBuilder.build();
} else {
logger.info(
'Found pre-generated docs for this entity. Serving them.',
);
// TODO: When to re-trigger build for cache invalidation?
}
}
}
res.redirect(`${storageUrl}${req.path.replace('/docs', '')}`);
});
// Route middleware which serves files from the storage set in the publisher.
router.use('/static/docs', publisher.docsRouter());
return router;
yarn.lock
+297 -7
View File
@@ -1828,6 +1828,65 @@
query-string "^6.13.3"
xcase "^2.0.1"
"@google-cloud/common@^3.5.0":
version "3.5.0"
resolved "https://registry.npmjs.org/@google-cloud/common/-/common-3.5.0.tgz#0959e769e8075a06eb0823cc567eef00fd0c2d02"
integrity sha512-10d7ZAvKhq47L271AqvHEd8KzJqGU45TY+rwM2Z3JHuB070FeTi7oJJd7elfrnKaEvaktw3hH2wKnRWxk/3oWQ==
dependencies:
"@google-cloud/projectify" "^2.0.0"
"@google-cloud/promisify" "^2.0.0"
arrify "^2.0.1"
duplexify "^4.1.1"
ent "^2.2.0"
extend "^3.0.2"
google-auth-library "^6.1.1"
retry-request "^4.1.1"
teeny-request "^7.0.0"
"@google-cloud/paginator@^3.0.0":
version "3.0.5"
resolved "https://registry.npmjs.org/@google-cloud/paginator/-/paginator-3.0.5.tgz#9d6b96c421a89bd560c1bc2c197c7611ef21db6c"
integrity sha512-N4Uk4BT1YuskfRhKXBs0n9Lg2YTROZc6IMpkO/8DIHODtm5s3xY8K5vVBo23v/2XulY3azwITQlYWgT4GdLsUw==
dependencies:
arrify "^2.0.0"
extend "^3.0.2"
"@google-cloud/projectify@^2.0.0":
version "2.0.1"
resolved "https://registry.npmjs.org/@google-cloud/projectify/-/projectify-2.0.1.tgz#13350ee609346435c795bbfe133a08dfeab78d65"
integrity sha512-ZDG38U/Yy6Zr21LaR3BTiiLtpJl6RkPS/JwoRT453G+6Q1DhlV0waNf8Lfu+YVYGIIxgKnLayJRfYlFJfiI8iQ==
"@google-cloud/promisify@^2.0.0":
version "2.0.3"
resolved "https://registry.npmjs.org/@google-cloud/promisify/-/promisify-2.0.3.tgz#f934b5cdc939e3c7039ff62b9caaf59a9d89e3a8"
integrity sha512-d4VSA86eL/AFTe5xtyZX+ePUjE8dIFu2T8zmdeNBSa5/kNgXPCx/o/wbFNHAGLJdGnk1vddRuMESD9HbOC8irw==
"@google-cloud/storage@^5.6.0":
version "5.6.0"
resolved "https://registry.npmjs.org/@google-cloud/storage/-/storage-5.6.0.tgz#bc6925c7970c375212a4da21c123298fc9665dec"
integrity sha512-nLcym8IuCzy1O7tNTXNFuMHfX900sTM3kSTqbKe7oFSoKUiaIM+FHuuuDimMMlieY6StA1xYNPRFFHz57Nv8YQ==
dependencies:
"@google-cloud/common" "^3.5.0"
"@google-cloud/paginator" "^3.0.0"
"@google-cloud/promisify" "^2.0.0"
arrify "^2.0.0"
compressible "^2.0.12"
date-and-time "^0.14.0"
duplexify "^4.0.0"
extend "^3.0.2"
gaxios "^4.0.0"
gcs-resumable-upload "^3.1.0"
get-stream "^6.0.0"
hash-stream-validation "^0.2.2"
mime "^2.2.0"
mime-types "^2.0.8"
onetime "^5.1.0"
p-limit "^3.0.1"
pumpify "^2.0.0"
snakeize "^0.1.0"
stream-events "^1.0.1"
xdg-basedir "^4.0.0"
"@graphql-codegen/cli@^1.17.7":
version "1.17.10"
resolved "https://registry.npmjs.org/@graphql-codegen/cli/-/cli-1.17.10.tgz#efebf9b887fdb94dd26dbf3eb1950e832efcda0e"
@@ -5492,6 +5551,13 @@
dependencies:
"@types/node" "*"
"@types/klaw@^3.0.1":
version "3.0.1"
resolved "https://registry.npmjs.org/@types/klaw/-/klaw-3.0.1.tgz#29f90021c0234976aa4eb97efced9cb6db9fa8b3"
integrity sha512-acnF3n9mYOr1aFJKFyvfNX0am9EtPUsYPq22QUCGdJE+MVt6UyAN1jwo+PmOPqXD4K7ZS9MtxDEp/un0lxFccA==
dependencies:
"@types/node" "*"
"@types/koa-compose@*":
version "3.2.5"
resolved "https://registry.npmjs.org/@types/koa-compose/-/koa-compose-3.2.5.tgz#85eb2e80ac50be95f37ccf8c407c09bbe3468e9d"
@@ -6521,6 +6587,13 @@ abbrev@1:
resolved "https://registry.npmjs.org/abbrev/-/abbrev-1.1.1.tgz#f8f2c887ad10bf67f634f005b6987fed3179aac8"
integrity sha512-nne9/IiQ/hzIhY6pdDnbBtz7DjPTKrY00P/zvPSm5pOFkl6xuGrGnXn/VtTNNfNtAfZ9/1RtehkszU9qcTii0Q==
abort-controller@^3.0.0:
version "3.0.0"
resolved "https://registry.npmjs.org/abort-controller/-/abort-controller-3.0.0.tgz#eaf54d53b62bae4138e809ca225c8439a6efb392"
integrity sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg==
dependencies:
event-target-shim "^5.0.0"
abstract-logging@^2.0.0:
version "2.0.0"
resolved "https://registry.npmjs.org/abstract-logging/-/abstract-logging-2.0.0.tgz#08a85814946c98ef06f4256ad470aba1886d4490"
@@ -7208,6 +7281,11 @@ arrify@^1.0.1:
resolved "https://registry.npmjs.org/arrify/-/arrify-1.0.1.tgz#898508da2226f380df904728456849c1501a4b0d"
integrity sha1-iYUI2iIm84DfkEcoRWhJwVAaSw0=
arrify@^2.0.0, arrify@^2.0.1:
version "2.0.1"
resolved "https://registry.npmjs.org/arrify/-/arrify-2.0.1.tgz#c9655e9331e0abcd588d2a7cad7e9956f66701fa"
integrity sha512-3duEwti880xqi4eAMN8AyR4a0ByT90zoYdLlevfrvU43vb0YZwZVfxOgxWrLXXXpyugL0hNZc9G6BiB5B3nUug==
asap@^2.0.0, asap@~2.0.3, asap@~2.0.6:
version "2.0.6"
resolved "https://registry.npmjs.org/asap/-/asap-2.0.6.tgz#e50347611d7e690943208bbdafebcbc2fb866d46"
@@ -7807,7 +7885,7 @@ base64-js@^1.0.2, base64-js@^1.2.0:
resolved "https://registry.npmjs.org/base64-js/-/base64-js-1.3.1.tgz#58ece8cb75dd07e71ed08c736abc5fac4dbf8df1"
integrity sha512-mLQ4i2QO1ytvGWFWmcngKO//JXAQueZvwEKtjgQFM4jIK0kU+ytMfplL8j+n5mspOfjHwoAg+9yhb7BwAHm36g==
base64-js@^1.3.1:
base64-js@^1.3.0, base64-js@^1.3.1:
version "1.5.1"
resolved "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz#1b1b440160a5bf7ad40b650f095963481903930a"
integrity sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==
@@ -7893,6 +7971,11 @@ big.js@^5.2.2:
resolved "https://registry.npmjs.org/big.js/-/big.js-5.2.2.tgz#65f0af382f578bcdc742bd9c281e9cb2d7768328"
integrity sha512-vyL2OymJxmarO8gxMr0mhChsO9QGwhynfuu4+MHTAW6czfq9humCB7rKpUjDd9YUiDPU4mzpyupFSvOClAwbmQ==
bignumber.js@^9.0.0:
version "9.0.1"
resolved "https://registry.npmjs.org/bignumber.js/-/bignumber.js-9.0.1.tgz#8d7ba124c882bfd8e43260c67475518d0689e4e5"
integrity sha512-IdZR9mh6ahOBv/hYGiXyVuyCetmGJhtYkqLBpTStdhEGjegpPlUawydyaF3pbIOFynJTpllEs+NP+CS9jKFLjA==
binary-extensions@^1.0.0:
version "1.13.1"
resolved "https://registry.npmjs.org/binary-extensions/-/binary-extensions-1.13.1.tgz#598afe54755b2868a5330d2aff9d4ebb53209b65"
@@ -9134,7 +9217,7 @@ compress-commons@^4.0.0:
normalize-path "^3.0.0"
readable-stream "^3.6.0"
compressible@~2.0.16:
compressible@^2.0.12, compressible@~2.0.16:
version "2.0.18"
resolved "https://registry.npmjs.org/compressible/-/compressible-2.0.18.tgz#af53cca6b070d4c3c0750fbd77286a6d7cc46fba"
integrity sha512-AF3r7P5dWxL8MxyITRMlORQNaOA2IkAFaTr4k7BUumjPtRpGDTZpl0Pb1XCO6JeDCBdp126Cgs9sMxqSjgYyRg==
@@ -9228,7 +9311,7 @@ config-chain@^1.1.11:
ini "^1.3.4"
proto-list "~1.2.1"
configstore@^5.0.1:
configstore@^5.0.0, configstore@^5.0.1:
version "5.0.1"
resolved "https://registry.npmjs.org/configstore/-/configstore-5.0.1.tgz#d365021b5df4b98cdd187d6a3b0e3f6a7cc5ed96"
integrity sha512-aMKprgk5YhBNyH25hj8wGt2+D52Sw1DRRIzqBwLp2Ya9mFmY8KPvvtvmna8SxVR9JMZ4kzMD68N22vlaRpkeFA==
@@ -10193,6 +10276,11 @@ dataloader@2.0.0:
resolved "https://registry.npmjs.org/dataloader/-/dataloader-2.0.0.tgz#41eaf123db115987e21ca93c005cd7753c55fe6f"
integrity sha512-YzhyDAwA4TaQIhM5go+vCLmU0UikghC/t9DTQYZR2M/UvZ1MdOhPezSDZcjj9uqQJOMqjLcpWtyW2iNINdlatQ==
date-and-time@^0.14.0:
version "0.14.1"
resolved "https://registry.npmjs.org/date-and-time/-/date-and-time-0.14.1.tgz#969634697b78956fb66b8be6fb0f39fbd631f2f6"
integrity sha512-M4RggEH5OF2ZuCOxgOU67R6Z9ohjKbxGvAQz48vj53wLmL0bAgumkBvycR32f30pK+Og9pIR+RFDyChbaE4oLA==
date-fns@^1.27.2:
version "1.30.1"
resolved "https://registry.npmjs.org/date-fns/-/date-fns-1.30.1.tgz#2e71bf0b119153dbb4cc4e88d9ea5acfb50dc05c"
@@ -10874,6 +10962,16 @@ duplexify@^3.4.2, duplexify@^3.6.0:
readable-stream "^2.0.0"
stream-shift "^1.0.0"
duplexify@^4.0.0, duplexify@^4.1.1:
version "4.1.1"
resolved "https://registry.npmjs.org/duplexify/-/duplexify-4.1.1.tgz#7027dc374f157b122a8ae08c2d3ea4d2d953aa61"
integrity sha512-DY3xVEmVHTv1wSzKNbwoU6nVjzI369Y6sPoqfYr0/xlx3IdX2n94xIszTcjPO8W8ZIv0Wb0PXNcjuZyT4wiICA==
dependencies:
end-of-stream "^1.4.1"
inherits "^2.0.3"
readable-stream "^3.1.1"
stream-shift "^1.0.0"
ecc-jsbn@~0.1.1:
version "0.1.2"
resolved "https://registry.npmjs.org/ecc-jsbn/-/ecc-jsbn-0.1.2.tgz#3a83a904e54353287874c564b7549386849a98c9"
@@ -10882,7 +10980,7 @@ ecc-jsbn@~0.1.1:
jsbn "~0.1.0"
safer-buffer "^2.1.0"
ecdsa-sig-formatter@1.0.11:
ecdsa-sig-formatter@1.0.11, ecdsa-sig-formatter@^1.0.11:
version "1.0.11"
resolved "https://registry.npmjs.org/ecdsa-sig-formatter/-/ecdsa-sig-formatter-1.0.11.tgz#ae0f0fa2d85045ef14a817daa3ce9acd0489e5bf"
integrity sha512-nagl3RYrbNv6kQkeJIpt6NJZy8twLB/2vtz6yN9Z4vRKHN4/QZJIEbqohALSgwKdnksuY3k5Addp5lg8sVoVcQ==
@@ -11026,6 +11124,11 @@ enquirer@^2.3.0, enquirer@^2.3.5:
dependencies:
ansi-colors "^4.1.1"
ent@^2.2.0:
version "2.2.0"
resolved "https://registry.npmjs.org/ent/-/ent-2.2.0.tgz#e964219325a21d05f44466a2f686ed6ce5f5dd1d"
integrity sha1-6WQhkyWiHQX0RGai9obtbOX13R0=
entities@^1.1.1, entities@^1.1.2:
version "1.1.2"
resolved "https://registry.npmjs.org/entities/-/entities-1.1.2.tgz#bdfa735299664dfafd34529ed4f8522a275fea56"
@@ -11574,6 +11677,11 @@ event-stream@=3.3.4:
stream-combiner "~0.0.4"
through "~2.3.1"
event-target-shim@^5.0.0:
version "5.0.1"
resolved "https://registry.npmjs.org/event-target-shim/-/event-target-shim-5.0.1.tgz#5d4d3ebdf9583d63a5333ce2deb7480ab2b05789"
integrity sha512-i/2XbnSz/uxRCU6+NdVJgKWDTM427+MqYbkQzD321DuCQJUqOuJKIA0IM2+W2xtYHdKOmZ4dR6fExsd4SXL+WQ==
eventemitter2@^6.4.2:
version "6.4.3"
resolved "https://registry.npmjs.org/eventemitter2/-/eventemitter2-6.4.3.tgz#35c563619b13f3681e7eb05cbdaf50f56ba58820"
@@ -11809,7 +11917,7 @@ extend-shallow@^3.0.0, extend-shallow@^3.0.2:
assign-symbols "^1.0.0"
is-extendable "^1.0.1"
extend@3.0.2, extend@^3.0.0, extend@~3.0.2:
extend@3.0.2, extend@^3.0.0, extend@^3.0.2, extend@~3.0.2:
version "3.0.2"
resolved "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz#f8b1136b4071fbd8eb140aff858b1019ec2915fa"
integrity sha512-fjquC59cD7CyW6urNXK0FBufkZcoiGG80wTuPujX590cB5Ttln20E2UB4S/WARVqhXffZl2LNgS+gQdPIIim/g==
@@ -11936,6 +12044,11 @@ fast-shallow-equal@^1.0.0:
resolved "https://registry.npmjs.org/fast-shallow-equal/-/fast-shallow-equal-1.0.0.tgz#d4dcaf6472440dcefa6f88b98e3251e27f25628b"
integrity sha512-HPtaa38cPgWvaCFmRNhlc6NG7pv6NUHqjPgVAkWGoB9mQMwYB27/K0CvOM5Czy+qpT3e8XJ6Q4aPAnzpNpzNaw==
fast-text-encoding@^1.0.0:
version "1.0.3"
resolved "https://registry.npmjs.org/fast-text-encoding/-/fast-text-encoding-1.0.3.tgz#ec02ac8e01ab8a319af182dae2681213cfe9ce53"
integrity sha512-dtm4QZH9nZtcDt8qJiOH9fcQd1NAgi+K1O2DbE6GG1PPCK/BWfOH3idCTRQ4ImXRUOyopDEgDEnVEE7Y/2Wrig==
fastest-stable-stringify@^1.0.1:
version "1.0.1"
resolved "https://registry.npmjs.org/fastest-stable-stringify/-/fastest-stable-stringify-1.0.1.tgz#9122d406d4c9d98bea644a6b6853d5874b87b028"
@@ -12532,6 +12645,49 @@ gauge@~2.7.3:
strip-ansi "^3.0.1"
wide-align "^1.1.0"
gaxios@^3.0.0:
version "3.2.0"
resolved "https://registry.npmjs.org/gaxios/-/gaxios-3.2.0.tgz#11b6f0e8fb08d94a10d4d58b044ad3bec6dd486a"
integrity sha512-+6WPeVzPvOshftpxJwRi2Ozez80tn/hdtOUag7+gajDHRJvAblKxTFSSMPtr2hmnLy7p0mvYz0rMXLBl8pSO7Q==
dependencies:
abort-controller "^3.0.0"
extend "^3.0.2"
https-proxy-agent "^5.0.0"
is-stream "^2.0.0"
node-fetch "^2.3.0"
gaxios@^4.0.0:
version "4.0.1"
resolved "https://registry.npmjs.org/gaxios/-/gaxios-4.0.1.tgz#bc7b205a89d883452822cc75e138620c35e3291e"
integrity sha512-jOin8xRZ/UytQeBpSXFqIzqU7Fi5TqgPNLlUsSB8kjJ76+FiGBfImF8KJu++c6J4jOldfJUtt0YmkRj2ZpSHTQ==
dependencies:
abort-controller "^3.0.0"
extend "^3.0.2"
https-proxy-agent "^5.0.0"
is-stream "^2.0.0"
node-fetch "^2.3.0"
gcp-metadata@^4.2.0:
version "4.2.1"
resolved "https://registry.npmjs.org/gcp-metadata/-/gcp-metadata-4.2.1.tgz#31849fbcf9025ef34c2297c32a89a1e7e9f2cd62"
integrity sha512-tSk+REe5iq/N+K+SK1XjZJUrFPuDqGZVzCy2vocIHIGmPlTGsa8owXMJwGkrXr73NO0AzhPW4MF2DEHz7P2AVw==
dependencies:
gaxios "^4.0.0"
json-bigint "^1.0.0"
gcs-resumable-upload@^3.1.0:
version "3.1.1"
resolved "https://registry.npmjs.org/gcs-resumable-upload/-/gcs-resumable-upload-3.1.1.tgz#67c766a0555d6a352f9651b7603337207167d0de"
integrity sha512-RS1osvAicj9+MjCc6jAcVL1Pt3tg7NK2C2gXM5nqD1Gs0klF2kj5nnAFSBy97JrtslMIQzpb7iSuxaG8rFWd2A==
dependencies:
abort-controller "^3.0.0"
configstore "^5.0.0"
extend "^3.0.2"
gaxios "^3.0.0"
google-auth-library "^6.0.0"
pumpify "^2.0.0"
stream-events "^1.0.4"
generic-names@^2.0.1:
version "2.0.1"
resolved "https://registry.npmjs.org/generic-names/-/generic-names-2.0.1.tgz#f8a378ead2ccaa7a34f0317b05554832ae41b872"
@@ -12636,6 +12792,11 @@ get-stream@^5.0.0, get-stream@^5.1.0:
dependencies:
pump "^3.0.0"
get-stream@^6.0.0:
version "6.0.0"
resolved "https://registry.npmjs.org/get-stream/-/get-stream-6.0.0.tgz#3e0012cb6827319da2706e601a1583e8629a6718"
integrity sha512-A1B3Bh1UmL0bidM/YX2NsCOTnGJePL9rO/M+Mw3m9f2gUpfokS0hi5Eah0WSUEWZdZhIZtMjkIYS7mDfOqNHbg==
get-value@^2.0.3, get-value@^2.0.6:
version "2.0.6"
resolved "https://registry.npmjs.org/get-value/-/get-value-2.0.6.tgz#dc15ca1c672387ca76bd37ac0a395ba2042a2c28"
@@ -12930,6 +13091,28 @@ good-listener@^1.2.2:
dependencies:
delegate "^3.1.2"
google-auth-library@^6.0.0, google-auth-library@^6.1.1:
version "6.1.3"
resolved "https://registry.npmjs.org/google-auth-library/-/google-auth-library-6.1.3.tgz#39d868140b70d0c4b32c6f6d8f4ccc1400d84dca"
integrity sha512-m9mwvY3GWbr7ZYEbl61isWmk+fvTmOt0YNUfPOUY2VH8K5pZlAIWJjxEi0PqR3OjMretyiQLI6GURMrPSwHQ2g==
dependencies:
arrify "^2.0.0"
base64-js "^1.3.0"
ecdsa-sig-formatter "^1.0.11"
fast-text-encoding "^1.0.0"
gaxios "^4.0.0"
gcp-metadata "^4.2.0"
gtoken "^5.0.4"
jws "^4.0.0"
lru-cache "^6.0.0"
google-p12-pem@^3.0.3:
version "3.0.3"
resolved "https://registry.npmjs.org/google-p12-pem/-/google-p12-pem-3.0.3.tgz#673ac3a75d3903a87f05878f3c75e06fc151669e"
integrity sha512-wS0ek4ZtFx/ACKYF3JhyGe5kzH7pgiQ7J5otlumqR9psmWMYc+U9cErKlCYVYHoUaidXHdZ2xbo34kB+S+24hA==
dependencies:
node-forge "^0.10.0"
got@^10.7.0:
version "10.7.0"
resolved "https://registry.npmjs.org/got/-/got-10.7.0.tgz#62889dbcd6cca32cd6a154cc2d0c6895121d091f"
@@ -13195,6 +13378,16 @@ growly@^1.3.0:
resolved "https://registry.npmjs.org/growly/-/growly-1.3.0.tgz#f10748cbe76af964b7c96c93c6bcc28af120c081"
integrity sha1-8QdIy+dq+WS3yWyTxrzCivEgwIE=
gtoken@^5.0.4:
version "5.1.0"
resolved "https://registry.npmjs.org/gtoken/-/gtoken-5.1.0.tgz#4ba8d2fc9a8459098f76e7e8fd7beaa39fda9fe4"
integrity sha512-4d8N6Lk8TEAHl9vVoRVMh9BNOKWVgl2DdNtr3428O75r3QFrF/a5MMu851VmK0AA8+iSvbwRv69k5XnMLURGhg==
dependencies:
gaxios "^4.0.0"
google-p12-pem "^3.0.3"
jws "^4.0.0"
mime "^2.2.0"
gud@^1.0.0:
version "1.0.0"
resolved "https://registry.npmjs.org/gud/-/gud-1.0.0.tgz#a489581b17e6a70beca9abe3ae57de7a499852c0"
@@ -13326,6 +13519,11 @@ hash-base@^3.0.0:
inherits "^2.0.1"
safe-buffer "^5.0.1"
hash-stream-validation@^0.2.2:
version "0.2.4"
resolved "https://registry.npmjs.org/hash-stream-validation/-/hash-stream-validation-0.2.4.tgz#ee68b41bf822f7f44db1142ec28ba9ee7ccb7512"
integrity sha512-Gjzu0Xn7IagXVkSu9cSFuK1fqzwtLwFhNhVL8IFJijRNMgUttFbBSIAzKuSIrsFMO1+g1RlsoN49zPIbwPDMGQ==
hash.js@^1.0.0, hash.js@^1.0.3:
version "1.1.7"
resolved "https://registry.npmjs.org/hash.js/-/hash.js-1.1.7.tgz#0babca538e8d4ee4a0f8988d68866537a003cf42"
@@ -13618,7 +13816,7 @@ http-proxy-agent@^2.1.0:
agent-base "4"
debug "3.1.0"
http-proxy-agent@^4.0.1:
http-proxy-agent@^4.0.0, http-proxy-agent@^4.0.1:
version "4.0.1"
resolved "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-4.0.1.tgz#8a8c8ef7f5932ccf953c296ca8291b95aa74aa3a"
integrity sha512-k0zdNgqWTGA6aeIRVpvfVob4fL52dTfaehylg0Y4UvSySvOq/Y+BOyPrgpUrA7HylqvU8vIZGsRuXmspskV0Tg==
@@ -15287,6 +15485,13 @@ jsesc@~0.5.0:
resolved "https://registry.npmjs.org/jsesc/-/jsesc-0.5.0.tgz#e7dee66e35d6fc16f710fe91d5cf69f70f08911d"
integrity sha1-597mbjXW/Bb3EP6R1c9p9w8IkR0=
json-bigint@^1.0.0:
version "1.0.0"
resolved "https://registry.npmjs.org/json-bigint/-/json-bigint-1.0.0.tgz#ae547823ac0cad8398667f8cd9ef4730f5b01ff1"
integrity sha512-SiPv/8VpZuWbvLSMtTDU8hEfrZWg/mH/nV/b4o0CYbSxu1UIQPLdwKOCIyLQX+VIPO5vrLX3i8qtqFyhdPSUSQ==
dependencies:
bignumber.js "^9.0.0"
json-buffer@3.0.0:
version "3.0.0"
resolved "https://registry.npmjs.org/json-buffer/-/json-buffer-3.0.0.tgz#5b1f397afc75d677bde8bcfc0e47e1f9a3d9a898"
@@ -15587,6 +15792,15 @@ jwa@^1.4.1:
ecdsa-sig-formatter "1.0.11"
safe-buffer "^5.0.1"
jwa@^2.0.0:
version "2.0.0"
resolved "https://registry.npmjs.org/jwa/-/jwa-2.0.0.tgz#a7e9c3f29dae94027ebcaf49975c9345593410fc"
integrity sha512-jrZ2Qx916EA+fq9cEAeCROWPTfCwi1IVHqT2tapuqLEVVDKFDENFw1oL+MwrTvH6msKxsd1YTDVw6uKEcsrLEA==
dependencies:
buffer-equal-constant-time "1.0.1"
ecdsa-sig-formatter "1.0.11"
safe-buffer "^5.0.1"
jws@^3.2.2:
version "3.2.2"
resolved "https://registry.npmjs.org/jws/-/jws-3.2.2.tgz#001099f3639468c9414000e99995fa52fb478304"
@@ -15595,6 +15809,14 @@ jws@^3.2.2:
jwa "^1.4.1"
safe-buffer "^5.0.1"
jws@^4.0.0:
version "4.0.0"
resolved "https://registry.npmjs.org/jws/-/jws-4.0.0.tgz#2d4e8cf6a318ffaa12615e9dec7e86e6c97310f4"
integrity sha512-KDncfTmOZoOMTFG4mBlG0qUIOlc03fmzH+ru6RgYVZhPkyiy/92Owlt/8UEN+a4TXR1FQetfIpJE8ApdvdVxTg==
dependencies:
jwa "^2.0.0"
safe-buffer "^5.0.1"
jwt-decode@2.2.0:
version "2.2.0"
resolved "https://registry.npmjs.org/jwt-decode/-/jwt-decode-2.2.0.tgz#7d86bd56679f58ce6a84704a657dd392bba81a79"
@@ -15657,6 +15879,13 @@ klaw@^1.0.0:
optionalDependencies:
graceful-fs "^4.1.9"
klaw@^3.0.0:
version "3.0.0"
resolved "https://registry.npmjs.org/klaw/-/klaw-3.0.0.tgz#b11bec9cf2492f06756d6e809ab73a2910259146"
integrity sha512-0Fo5oir+O9jnXu5EefYbVK+mHMBeEVEy2cmctR1O1NECcCkPRreJKrS6Qt/j3KC2C148Dfo9i3pCmCMsdqGr0g==
dependencies:
graceful-fs "^4.1.9"
kleur@^3.0.3:
version "3.0.3"
resolved "https://registry.npmjs.org/kleur/-/kleur-3.0.3.tgz#a79c9ecc86ee1ce3fa6206d1216c501f147fc07e"
@@ -16862,7 +17091,7 @@ mime-db@1.44.0, "mime-db@>= 1.43.0 < 2":
resolved "https://registry.npmjs.org/mime-db/-/mime-db-1.44.0.tgz#fa11c5eb0aca1334b4233cb4d52f10c5a6272f92"
integrity sha512-/NOTfLrsPBVeH7YtFPgsVWveuL+4SjjYxaQ1xtM1KMFj7HdxlBlxeyNLzhyJVx7r4rZGJAZ/6lkKCitSc/Nmpg==
mime-types@^2.1.12, mime-types@^2.1.26, mime-types@~2.1.17, mime-types@~2.1.19, mime-types@~2.1.24:
mime-types@^2.0.8, mime-types@^2.1.12, mime-types@^2.1.26, mime-types@~2.1.17, mime-types@~2.1.19, mime-types@~2.1.24:
version "2.1.27"
resolved "https://registry.npmjs.org/mime-types/-/mime-types-2.1.27.tgz#47949f98e279ea53119f5722e0f34e529bec009f"
integrity sha512-JIhqnCasI9yD+SsmkquHBxTSEuZdQX5BuQnS2Vc7puQQQ+8yiP5AY5uWhpdv4YL4VM5c6iliiYWPgJ/nJQLp7w==
@@ -16874,6 +17103,11 @@ mime@1.6.0, mime@^1.4.1:
resolved "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz#32cd9e5c64553bd58d19a568af452acff04981b1"
integrity sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==
mime@^2.2.0:
version "2.4.6"
resolved "https://registry.npmjs.org/mime/-/mime-2.4.6.tgz#e5b407c90db442f2beb5b162373d07b69affa4d1"
integrity sha512-RZKhC3EmpBchfTGBVb8fb+RL2cWyw/32lshnsETttkBAyAUXSGHxbEJWWRXc751DrIxG1q04b8QwMbAwkRPpUA==
mime@^2.3.1, mime@^2.4.4:
version "2.4.4"
resolved "https://registry.npmjs.org/mime/-/mime-2.4.4.tgz#bd7b91135fc6b01cde3e9bae33d659b63d8857e5"
@@ -18177,6 +18411,13 @@ p-limit@^2.0.0, p-limit@^2.2.0:
dependencies:
p-try "^2.0.0"
p-limit@^3.0.1:
version "3.1.0"
resolved "https://registry.npmjs.org/p-limit/-/p-limit-3.1.0.tgz#e1daccbe78d0d1388ca18c64fea38e3e57e3706b"
integrity sha512-TYOanM3wGwNGsZN2cVTYPArw454xnXj5qmWF1bEoAc4+cU/ol7GVh7odevjp1FNHduHc3KZMcFduxU5Xc6uJRQ==
dependencies:
yocto-queue "^0.1.0"
p-locate@^2.0.0:
version "2.0.0"
resolved "https://registry.npmjs.org/p-locate/-/p-locate-2.0.0.tgz#20a0103b222a70c8fd39cc2e580680f3dde5ec43"
@@ -19707,6 +19948,15 @@ pumpify@^1.3.3:
inherits "^2.0.3"
pump "^2.0.0"
pumpify@^2.0.0:
version "2.0.1"
resolved "https://registry.npmjs.org/pumpify/-/pumpify-2.0.1.tgz#abfc7b5a621307c728b551decbbefb51f0e4aa1e"
integrity sha512-m7KOje7jZxrmutanlkS1daj1dS6z6BgslzOXmcSEpIlCxM3VJH7lG5QLeck/6hgF6F4crFf01UtQmNsJfweTAw==
dependencies:
duplexify "^4.1.1"
inherits "^2.0.3"
pump "^3.0.0"
punycode@1.3.2:
version "1.3.2"
resolved "https://registry.npmjs.org/punycode/-/punycode-1.3.2.tgz#9653a036fb7c1ee42342f2325cceefea3926c48d"
@@ -21183,6 +21433,13 @@ ret@~0.1.10:
resolved "https://registry.npmjs.org/ret/-/ret-0.1.15.tgz#b8a4825d5bdb1fc3f6f53c2bc33f81388681c7bc"
integrity sha512-TTlYpa+OL+vMMNG24xSlQGEJ3B/RzEfUlLct7b5G/ytav+wPrplCpVMFuwzXbkecJrb6IYo1iFb0S9v37754mg==
retry-request@^4.1.1:
version "4.1.3"
resolved "https://registry.npmjs.org/retry-request/-/retry-request-4.1.3.tgz#d5f74daf261372cff58d08b0a1979b4d7cab0fde"
integrity sha512-QnRZUpuPNgX0+D1xVxul6DbJ9slvo4Rm6iV/dn63e048MvGbUZiKySVt6Tenp04JqmchxjiLltGerOJys7kJYQ==
dependencies:
debug "^4.1.1"
retry@0.12.0, retry@^0.12.0:
version "0.12.0"
resolved "https://registry.npmjs.org/retry/-/retry-0.12.0.tgz#1b42a6266a21f07421d1b0b54b7dc167b01c013b"
@@ -21847,6 +22104,11 @@ smartwrap@^1.2.3:
wcwidth "^1.0.1"
yargs "^15.1.0"
snakeize@^0.1.0:
version "0.1.0"
resolved "https://registry.npmjs.org/snakeize/-/snakeize-0.1.0.tgz#10c088d8b58eb076b3229bb5a04e232ce126422d"
integrity sha1-EMCI2LWOsHazIpu1oE4jLOEmQi0=
snapdragon-node@^2.0.1:
version "2.1.1"
resolved "https://registry.npmjs.org/snapdragon-node/-/snapdragon-node-2.1.1.tgz#6c175f86ff14bdb0724563e8f3c1b021a286853b"
@@ -22298,6 +22560,13 @@ stream-each@^1.1.0:
end-of-stream "^1.1.0"
stream-shift "^1.0.0"
stream-events@^1.0.1, stream-events@^1.0.4, stream-events@^1.0.5:
version "1.0.5"
resolved "https://registry.npmjs.org/stream-events/-/stream-events-1.0.5.tgz#bbc898ec4df33a4902d892333d47da9bf1c406d5"
integrity sha512-E1GUzBSgvct8Jsb3v2X15pjzN1tYebtbLaMg+eBOUOAxgbLoSbT2NS91ckc5lJD1KfLjId+jXJRgo0qnV5Nerg==
dependencies:
stubs "^3.0.0"
stream-http@^2.7.2:
version "2.8.3"
resolved "https://registry.npmjs.org/stream-http/-/stream-http-2.8.3.tgz#b2d242469288a5a27ec4fe8933acf623de6514fc"
@@ -22577,6 +22846,11 @@ strong-log-transformer@^2.0.0:
minimist "^1.2.0"
through "^2.3.4"
stubs@^3.0.0:
version "3.0.0"
resolved "https://registry.npmjs.org/stubs/-/stubs-3.0.0.tgz#e8d2ba1fa9c90570303c030b6900f7d5f89abe5b"
integrity sha1-6NK6H6nJBXAwPAMLaQD31fiavls=
style-inject@^0.3.0:
version "0.3.0"
resolved "https://registry.npmjs.org/style-inject/-/style-inject-0.3.0.tgz#d21c477affec91811cc82355832a700d22bf8dd3"
@@ -22953,6 +23227,17 @@ tdigest@^0.1.1:
dependencies:
bintrees "1.0.1"
teeny-request@^7.0.0:
version "7.0.1"
resolved "https://registry.npmjs.org/teeny-request/-/teeny-request-7.0.1.tgz#bdd41fdffea5f8fbc0d29392cb47bec4f66b2b4c"
integrity sha512-sasJmQ37klOlplL4Ia/786M5YlOcoLGQyq2TE4WHSRupbAuDaQW0PfVxV4MtdBtRJ4ngzS+1qim8zP6Zp35qCw==
dependencies:
http-proxy-agent "^4.0.0"
https-proxy-agent "^5.0.0"
node-fetch "^2.6.1"
stream-events "^1.0.5"
uuid "^8.0.0"
telejson@^5.0.2:
version "5.0.2"
resolved "https://registry.npmjs.org/telejson/-/telejson-5.0.2.tgz#ed1e64be250cc1c757a53c19e1740b49832b3d51"
@@ -25045,6 +25330,11 @@ yn@^4.0.0:
resolved "https://registry.npmjs.org/yn/-/yn-4.0.0.tgz#611480051ea43b510da1dfdbe177ed159f00a979"
integrity sha512-huWiiCS4TxKc4SfgmTwW1K7JmXPPAmuXWYy4j9qjQo4+27Kni8mGhAAi1cloRWmBe2EqcLgt3IGqQoRL/MtPgg==
yocto-queue@^0.1.0:
version "0.1.0"
resolved "https://registry.npmjs.org/yocto-queue/-/yocto-queue-0.1.0.tgz#0294eb3dee05028d31ee1a5fa2c556a6aaf10a1b"
integrity sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q==
yup@^0.29.3:
version "0.29.3"
resolved "https://registry.npmjs.org/yup/-/yup-0.29.3.tgz#69a30fd3f1c19f5d9e31b1cf1c2b851ce8045fea"