diff --git a/.changeset/friendly-carpets-repeat.md b/.changeset/friendly-carpets-repeat.md new file mode 100644 index 0000000000..7fb0d2f6eb --- /dev/null +++ b/.changeset/friendly-carpets-repeat.md @@ -0,0 +1,45 @@ +--- +'@backstage/techdocs-common': minor +'@backstage/plugin-techdocs': minor +'@backstage/plugin-techdocs-backend': minor +--- + +_Breaking changes_ + +1. Added option to use Google Cloud Storage as a choice to store the static generated files for TechDocs. + It can be configured using `techdocs.publisher.type` option in `app-config.yaml`. + Step-by-step guide to configure GCS is available here https://backstage.io/docs/features/techdocs/using-cloud-storage + Set `techdocs.publisher.type` to `'local'` if you want to continue using local filesystem to store TechDocs files. + +2. `techdocs.builder` is now required and can be set to `'local'` or `'external'`. (Set it to `'local'` for now, since CI/CD build + workflow for TechDocs will be available soon (in few weeks)). + If builder is set to 'local' and you open a TechDocs page, `techdocs-backend` will try to generate the docs, publish to storage and + show the generated docs afterwords. + If builder is set to `'external'`, `techdocs-backend` will only fetch the docs and will NOT try to generate and publish. In this case of `'external'`, + we assume that docs are being built in the CI/CD pipeline of the repository. + TechDocs will not assume a default value for `techdocs.builder`. It is better to explicitly define it in the `app-config.yaml`. + +3. When configuring TechDocs in your backend, there is a difference in how a new publisher is created. + +``` +--- const publisher = new LocalPublish(logger, discovery); ++++ const publisher = Publisher.fromConfig(config, logger, discovery); +``` + +Based on the config `techdocs.publisher.type`, the publisher could be either Local publisher or Google Cloud Storage publisher. + +4. `techdocs.storageUrl` is now a required config. Should be `http://localhost:7000/api/techdocs/static/docs` in most setups. + +5. Parts of `@backstage/plugin-techdocs-backend` have been moved to a new package `@backstage/techdocs-common` to generate docs. Also to publish docs + to-and-fro between TechDocs and a storage (either local or external). However, a Backstage app does NOT need to import the `techdocs-common` package - + app should only import `@backstage/plugin-techdocs` and `@backstage/plugin-techdocs-backend`. + +_Patch changes_ + +1. See all of TechDocs config options and its documentation https://backstage.io/docs/features/techdocs/configuration + +2. Logic about serving static files and metadata retrieval have been abstracted away from the router in `techdocs-backend` to the instance of publisher. + +3. Removed Material UI Spinner from TechDocs header. Spinners cause unnecessary UX distraction. + Case 1 (when docs are built and are to be served): Spinners appear for a split second before the name of site shows up. This unnecessarily distracts eyes because spinners increase the size of the Header. A dot (.) would do fine. Definitely more can be done. + Case 2 (when docs are being generated): There is already a linear progress bar (which is recommended in Storybook). diff --git a/.changeset/six-mugs-camp.md b/.changeset/six-mugs-camp.md new file mode 100644 index 0000000000..373edd340a --- /dev/null +++ b/.changeset/six-mugs-camp.md @@ -0,0 +1,20 @@ +--- +'@backstage/create-app': patch +--- + +In the techdocs-backend plugin (`packages/backend/src/plugins/techdocs.ts`), create a publisher using + +``` + const publisher = Publisher.fromConfig(config, logger, discovery); +``` + +instead of + +``` + const publisher = new LocalPublish(logger, discovery); +``` + +An instance of `publisher` can either be a local filesystem publisher or a Google Cloud Storage publisher. + +Read more about the configs here https://backstage.io/docs/features/techdocs/configuration +(You will also have to update `techdocs.storage.type` to `local` or `googleGcs`. And `techdocs.builder` to either `local` or `external`.) diff --git a/.github/styles/vocab.txt b/.github/styles/vocab.txt index a4dbe761a6..5008e88dae 100644 --- a/.github/styles/vocab.txt +++ b/.github/styles/vocab.txt @@ -23,6 +23,7 @@ changesets Changesets chanwit Chanwit +ci cisphobia cissexist classname diff --git a/app-config.yaml b/app-config.yaml index 0eb00ebbf3..f6f71478d0 100644 --- a/app-config.yaml +++ b/app-config.yaml @@ -68,11 +68,15 @@ proxy: organization: name: My Company +# Reference documentation http://backstage.io/docs/features/techdocs/configuration techdocs: - storageUrl: http://localhost:7000/api/techdocs/static/docs requestUrl: http://localhost:7000/api/techdocs + storageUrl: http://localhost:7000/api/techdocs/static/docs + builder: 'local' # Alternatives - 'external' generators: - techdocs: 'docker' + techdocs: 'docker' # Alternatives - 'local' + publisher: + type: 'local' # Alternatives - 'googleGcs'. Read documentation for using alternatives. sentry: organization: my-company diff --git a/docs/features/techdocs/architecture.md b/docs/features/techdocs/architecture.md index 9305239b7f..3969002a46 100644 --- a/docs/features/techdocs/architecture.md +++ b/docs/features/techdocs/architecture.md @@ -50,8 +50,8 @@ built. We assume each entity lives in a repository somewhere (GitHub, GitLab, etc.). We recommend using a CI/CD pipeline with the repository that has a dedicated -step/job to build docs for TechDocs. The generated static files are then stored -in a cloud storage solution of your choice. +step/job to generate docs for TechDocs. The generated static files are then +stored in a cloud storage solution of your choice. [Track progress here](https://github.com/backstage/backstage/issues/3096). Similar to how it is done in the Basic setup, the TechDocs Reader requests @@ -60,10 +60,10 @@ your configured storage solution for the necessary files and returns them to TechDocs Reader. We will provide instructions, scripts and/or templates (e.g. GitHub Actions) to -build docs in your CI/CD system. +generate docs in your CI/CD system. [Track progress here.](https://github.com/backstage/backstage/issues/3400) You -will be able to use `techdocs-cli` to build docs and publish the generated docs -site files to your cloud storage system. +will be able to use `techdocs-cli` to generate docs and publish the generated +docs site files to your cloud storage system. Note about caching: We have noticed internally that some storage providers can be quite slow, which is why we are recommending a cache that sits between the @@ -120,8 +120,8 @@ docs site in real-time?** A: Generating the content from Markdown on the fly is not optimal (although that is how the basic out-of-the-box setup is implemented). Storage solutions act as a cache for the generated static content. TechDocs is also currently built on -MkDocs which does not allow us to build docs per-page, so we would have to build -all docs for a entity on every request. +MkDocs which does not allow us to generate docs per-page, so we would have to +build all docs for a entity on every request. # Future work diff --git a/docs/features/techdocs/configuration.md b/docs/features/techdocs/configuration.md new file mode 100644 index 0000000000..b3cb349778 --- /dev/null +++ b/docs/features/techdocs/configuration.md @@ -0,0 +1,73 @@ +--- +id: configuration +title: TechDocs Configuration Options +description: + Reference documentation for configuring TechDocs using app-config.yaml +--- + +Using the `app-config.yaml` in the Backstage app, you can configure TechDocs +using several options. This page serves as a reference to all the available +configuration options for TechDocs. + +```yaml +# File: app-config.yaml + +techdocs: + + # TechDocs makes API calls to techdocs-backend using this URL. e.g. get docs of an entity, get metadata, etc. + + requestUrl: http://localhost:7000/api/techdocs + + + # Just another route in techdocs-backend where TechDocs requests the static files from. This URL uses an HTTP middleware + # to serve files from either a local directory or an External storage provider. + + storageUrl: http://localhost:7000/api/techdocs/static/docs + + + # generators.techdocs can have two values: 'docker' or 'local'. This is to determine how to run the generator - whether to + # spin up the techdocs-container docker image or to run mkdocs locally (assuming all the dependencies are taken care of). + # You want to change this to 'local' if you are running Backstage using your own custom Docker setup and want to avoid running + # into Docker in Docker situation. Read more here + # https://backstage.io/docs/features/techdocs/getting-started#disable-docker-in-docker-situation-optional + + generators: + techdocs: 'docker' + + + # techdocs.builder can be either 'local' or 'external. + # If builder is set to 'local' and you open a TechDocs page, techdocs-backend will try to generate the docs, publish to storage + # and show the generated docs afterwords. This is the "Basic" setup of the TechDocs Architecture. + # If builder is set to 'external', techdocs-backend will only fetch the docs and will NOT try to generate and publish. In this case of 'external', + # we assume that docs are being built by an external process (e.g. in the CI/CD pipeline of the repository). This is the "Recommended" setup of + # the architecture. Read more here https://backstage.io/docs/features/techdocs/architecture + + builder: 'local' + + + # techdocs.publisher is used to configure the Storage option, whether you want to use the local filesystem to store generated docs + # or you want to use External storage providers like Google Cloud Storage, AWS S3, etc. + + publisher: + + # techdocs.publisher.type can be - 'local' or 'googleGcs' (awsS3, azureStorage, etc. to be available as well). + # When set to 'local', techdocs-backend will create a 'static' directory at its root to store generated documentation files. + # When set to 'googleGcs', techdocs-backend will use a Google Cloud Storage Bucket to store generated documentation files. + + type: 'local' + + + # Required when techdocs.publisher.type is set to 'googleGcs'. Skip otherwise. + + googleGcs: + # An API key is required to write to a storage bucket. + credentials: + $file: '/path/to/google_application_credentials.json', + + # Your GCP Project ID where the Cloud Storage Bucket is hosted. + projectId: 'gcp-project-id' + + # Cloud Storage Bucket Name + bucketName: 'techdocs-storage', + +``` diff --git a/docs/features/techdocs/using-cloud-storage.md b/docs/features/techdocs/using-cloud-storage.md new file mode 100644 index 0000000000..55352b6c00 --- /dev/null +++ b/docs/features/techdocs/using-cloud-storage.md @@ -0,0 +1,95 @@ +--- +id: using-cloud-storage +title: Using Cloud Storage for TechDocs generated files +description: Using Cloud Storage for TechDocs generated files +--- + +In the [TechDocs architecture](./architecture.md) you have the option to choose +where you want to store the Generated static files which TechDocs uses to render +documentation. In both the "Basic" and "Recommended" setup, you can add cloud +storage providers like Google GCS, Amazon AWS S3, etc. By default, TechDocs uses +the local filesystem of the `techdocs-backend` plugin in the "Basic" setup. And +in the recommended setup, having one of the cloud storage is a prerequisite. +Read more on the TechDocs Architecture documentation page. + +On this page you can read how to enable them. + +## Configuring Google GCS Bucket with TechDocs + +Follow the +[official Google Cloud documentation](https://googleapis.dev/nodejs/storage/latest/index.html#quickstart) +for the latest instructions on the following steps involving GCP. + +**1. Set `techdocs.publisher.type` config in your `app-config.yaml`** + +Set `techdocs.publisher.type` to `'googleGcs'`. + +```yaml +techdocs: + publisher: + type: 'googleGcs' +``` + +**2. GCP (Google Cloud Platform) Project** + +Create or choose a dedicated GCP project. Set +`techdocs.publisher.googleGcs.projectId` to the project ID. + +```yaml +techdocs: + publisher: + type: 'googleGcs' + googleGcs: + projectId: 'gcp-project-id +``` + +**3. Service account API key** + +Create a new Service Account and a key associated with it. In roles of the +service account, use "Storage Admin". + +If you want to create a custom role, make sure to include both `get` and +`create` permissions for both "Objects" and "Buckets". See +https://cloud.google.com/storage/docs/access-control/iam-permissions + +A service account can have many keys. Open your newly created account's page (in +IAM & Admin console), and create a new key. Use JSON format for the key. + +A `.json` file will be downloaded. This is the secret +key TechDocs will use to make API calls. Make it available in your Backstage +server and/or your local development server and set it in the app config +`techdocs.publisher.googleGcs.credentials`. + +```yaml +techdocs: + publisher: + type: 'googleGcs' + googleGcs: + projectId: 'gcp-project-id' + credentials: + $file: '/path/to/google_application_credentials.json' +``` + +**4. GCS Bucket** + +Create a dedicated bucket for TechDocs sites. techdocs-backend will publish +documentation to this bucket. TechDocs will fetch files from here to serve +documentation in Backstage. + +Set the name of the bucket to `techdocs.publisher + +```yaml +techdocs: + publisher: + type: 'googleGcs' + googleGcs: + projectId: 'gcp-project-id' + credentials: + $file: '/path/to/google_application_credentials.json' + bucketName: 'name-of-techdocs-storage-bucket' +``` + +**5. That's it!** + +Your Backstage app is now ready to use Google Cloud Storage for TechDocs, to +store the static generated documentation files. diff --git a/microsite/sidebars.json b/microsite/sidebars.json index e60d9c520e..91112dd3b3 100644 --- a/microsite/sidebars.json +++ b/microsite/sidebars.json @@ -74,6 +74,8 @@ "features/techdocs/concepts", "features/techdocs/architecture", "features/techdocs/creating-and-publishing", + "features/techdocs/configuration", + "features/techdocs/using-cloud-storage", "features/techdocs/troubleshooting", "features/techdocs/faqs" ] diff --git a/mkdocs.yml b/mkdocs.yml index 72508c821e..109be46a58 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -54,6 +54,8 @@ nav: - Concepts: 'features/techdocs/concepts.md' - TechDocs Architecture: 'features/techdocs/architecture.md' - Creating and Publishing Documentation: 'features/techdocs/creating-and-publishing.md' + - Configuration: 'features/techdocs/configuration.md' + - Using Cloud Storage: 'features/techdocs/using-cloud-storage.md' - Troubleshooting: 'features/techdocs/troubleshooting.md' - FAQ: 'features/techdocs/FAQ.md' - Plugins: diff --git a/packages/backend/src/plugins/techdocs.ts b/packages/backend/src/plugins/techdocs.ts index de48280e64..4720b2568e 100644 --- a/packages/backend/src/plugins/techdocs.ts +++ b/packages/backend/src/plugins/techdocs.ts @@ -13,16 +13,15 @@ * See the License for the specific language governing permissions and * limitations under the License. */ - import { createRouter, DirectoryPreparer, Preparers, Generators, - LocalPublish, TechdocsGenerator, CommonGitPreparer, UrlPreparer, + Publisher, } from '@backstage/plugin-techdocs-backend'; import { PluginEnvironment } from '../types'; import Docker from 'dockerode'; @@ -50,7 +49,7 @@ export default async function createPlugin({ const urlPreparer = new UrlPreparer(reader, logger); preparers.register('url', urlPreparer); - const publisher = new LocalPublish(logger, discovery); + const publisher = Publisher.fromConfig(config, logger, discovery); const dockerClient = new Docker(); diff --git a/packages/create-app/templates/default-app/app-config.yaml.hbs b/packages/create-app/templates/default-app/app-config.yaml.hbs index 595812dec3..7b70a2192c 100644 --- a/packages/create-app/templates/default-app/app-config.yaml.hbs +++ b/packages/create-app/templates/default-app/app-config.yaml.hbs @@ -57,10 +57,13 @@ proxy: changeOrigin: true techdocs: - storageUrl: http://localhost:7000/api/techdocs/static/docs requestUrl: http://localhost:7000/api/techdocs + storageUrl: http://localhost:7000/api/techdocs/static/docs + builder: 'local' generators: techdocs: 'docker' + publisher: + type: 'local' lighthouse: baseUrl: http://localhost:3003 diff --git a/packages/create-app/templates/default-app/packages/backend/src/plugins/techdocs.ts b/packages/create-app/templates/default-app/packages/backend/src/plugins/techdocs.ts index ac4d81a8e8..1bbb5ff24b 100644 --- a/packages/create-app/templates/default-app/packages/backend/src/plugins/techdocs.ts +++ b/packages/create-app/templates/default-app/packages/backend/src/plugins/techdocs.ts @@ -1,11 +1,11 @@ import { createRouter, DirectoryPreparer, - CommonGitPreparer, Preparers, Generators, - LocalPublish, TechdocsGenerator, + CommonGitPreparer, + Publisher, } from '@backstage/plugin-techdocs-backend'; import { PluginEnvironment } from '../types'; import Docker from 'dockerode'; @@ -28,7 +28,7 @@ export default async function createPlugin({ preparers.register('github', commonGitPreparer); preparers.register('gitlab', commonGitPreparer); - const publisher = new LocalPublish(logger, discovery); + const publisher = Publisher.fromConfig(config, logger, discovery); const dockerClient = new Docker(); diff --git a/packages/techdocs-common/.eslintrc.js b/packages/techdocs-common/.eslintrc.js new file mode 100644 index 0000000000..16a033dbc6 --- /dev/null +++ b/packages/techdocs-common/.eslintrc.js @@ -0,0 +1,3 @@ +module.exports = { + extends: [require.resolve('@backstage/cli/config/eslint.backend')], +}; diff --git a/packages/techdocs-common/README.md b/packages/techdocs-common/README.md new file mode 100644 index 0000000000..e4889d6f79 --- /dev/null +++ b/packages/techdocs-common/README.md @@ -0,0 +1,49 @@ +# @backstage/techdocs-common + +Common functionalities for TechDocs, to be shared between techdocs-backend plugin and techdocs-cli + +This package is used by `techdocs-backend` to serve docs from different types of publishers (Google GCS, Local, etc.). +It is also used to build docs and publish them to storage, by both `techdocs-backend` and `techdocs-cli`. + +## Usage + +Create a preparer instance from the [preparers available](/packages/techdocs-common/src/stages/prepare) at which takes an Entity instance. +Run the [docs generator](/packages/techdocs-common/src/stages/generate) on the prepared directory. +Publish the generated directory files to a [storage](/packages/techdocs-common/src/stages/publish) of your choice. + +Example: + +```js +async () => { + const preparedDir = await preparer.prepare(entity); + + const parsedLocationAnnotation = getLocationForEntity(entity); + const { resultDir } = await generator.run({ + directory: preparedDir, + dockerClient: dockerClient, + parsedLocationAnnotation, + }); + + await publisher.publish({ + entity: entity, + directory: resultDir, + }); +}; +``` + +## Features + +Currently the build process is split up in these three stages. + +- Preparers +- Generators +- Publishers + +Preparers read your entity data and creates a working directory with your documentation source code. For example if you have set your `backstage.io/techdocs-ref` to `github:https://github.com/backstage/backstage.git` it will clone that repository to a temp folder and pass that on to the generator. + +Generators takes the prepared source and runs the `techdocs-container` on it. It then passes on the output folder of that build to the publisher. + +Publishers gets a folder path from the generator and publish it to your storage solution. Read documentation to know more about configuring storage solutions. +http://backstage.io/docs/features/techdocs/configuration + +Any of these can be extended. We want to extend our support to most of the storage providers (Publishers) and source code host providers (Preparers). diff --git a/packages/techdocs-common/__mocks__/@google-cloud/storage.ts b/packages/techdocs-common/__mocks__/@google-cloud/storage.ts new file mode 100644 index 0000000000..e95cee11d0 --- /dev/null +++ b/packages/techdocs-common/__mocks__/@google-cloud/storage.ts @@ -0,0 +1,53 @@ +/* + * Copyright 2020 Spotify AB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +type storageOptions = { + projectId?: string; + keyFilename?: string; +}; + +class Bucket { + private readonly bucketName; + + constructor(bucketName: string) { + this.bucketName = bucketName; + } + + getMetadata() { + return new Promise(resolve => { + resolve(''); + }); + } + + upload(source: string, { destination }) { + return new Promise(resolve => { + resolve({ source, destination }); + }); + } +} + +export class Storage { + private readonly projectId; + private readonly keyFilename; + + constructor(options: storageOptions) { + this.projectId = options.projectId; + this.keyFilename = options.keyFilename; + } + + bucket(bucketName) { + return new Bucket(bucketName); + } +} diff --git a/packages/techdocs-common/package.json b/packages/techdocs-common/package.json new file mode 100644 index 0000000000..bee9bfdf93 --- /dev/null +++ b/packages/techdocs-common/package.json @@ -0,0 +1,64 @@ +{ + "name": "@backstage/techdocs-common", + "description": "Common functionalities for TechDocs, to be shared between techdocs-backend plugin and techdocs-cli", + "version": "0.1.1", + "main": "src/index.ts", + "types": "src/index.ts", + "private": false, + "publishConfig": { + "access": "public", + "main": "dist/index.cjs.js", + "types": "dist/index.d.ts" + }, + "homepage": "https://backstage.io", + "repository": { + "type": "git", + "url": "https://github.com/backstage/backstage", + "directory": "packages/techdocs-common" + }, + "keywords": [ + "techdocs", + "backstage" + ], + "license": "Apache-2.0", + "files": [ + "dist" + ], + "scripts": { + "build": "backstage-cli build --outputs cjs,types", + "lint": "backstage-cli lint", + "test": "backstage-cli test", + "prepack": "backstage-cli prepack", + "postpack": "backstage-cli postpack", + "clean": "backstage-cli clean" + }, + "bugs": { + "url": "https://github.com/backstage/backstage/issues" + }, + "dependencies": { + "@backstage/backend-common": "^0.4.0", + "@backstage/catalog-model": "^0.5.0", + "@backstage/config": "^0.1.2", + "@google-cloud/storage": "^5.6.0", + "@types/dockerode": "^3.2.1", + "@types/express": "^4.17.6", + "cross-fetch": "^3.0.6", + "dockerode": "^3.2.1", + "express": "^4.17.1", + "fs-extra": "^9.0.1", + "git-url-parse": "^11.4.0", + "js-yaml": "^3.14.0", + "mock-fs": "^4.13.0", + "nodegit": "^0.27.0", + "recursive-readdir": "^2.2.2", + "winston": "^3.2.1" + }, + "devDependencies": { + "@backstage/cli": "^0.4.1" + }, + "jest": { + "roots": [ + ".." + ] + } +} diff --git a/plugins/techdocs-backend/src/default-branch.ts b/packages/techdocs-common/src/default-branch.ts similarity index 100% rename from plugins/techdocs-backend/src/default-branch.ts rename to packages/techdocs-common/src/default-branch.ts diff --git a/plugins/techdocs-backend/src/git-auth.ts b/packages/techdocs-common/src/git-auth.ts similarity index 100% rename from plugins/techdocs-backend/src/git-auth.ts rename to packages/techdocs-common/src/git-auth.ts diff --git a/packages/techdocs-common/src/helpers.test.ts b/packages/techdocs-common/src/helpers.test.ts new file mode 100644 index 0000000000..88c4d8829e --- /dev/null +++ b/packages/techdocs-common/src/helpers.test.ts @@ -0,0 +1,149 @@ +/* + * Copyright 2020 Spotify AB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { Readable } from 'stream'; +import { + getDocFilesFromRepository, + getLocationForEntity, + parseReferenceAnnotation, +} from './helpers'; +import { UrlReader, ReadTreeResponse } from '@backstage/backend-common'; +import { Entity } from '@backstage/catalog-model'; + +const entityBase: Entity = { + metadata: { + namespace: 'default', + name: 'mytestcomponent', + description: 'A component for testing', + }, + apiVersion: 'backstage.io/v1alpha1', + kind: 'Component', + spec: { + type: 'documentation', + lifecycle: 'experimental', + owner: 'testuser', + }, +}; + +const metadataBase = { + namespace: 'default', + name: 'mytestcomponent', + description: 'A component for testing', +}; + +const goodAnnotation = { + annotations: { + 'backstage.io/techdocs-ref': + 'url:https://github.com/backstage/backstage/blob/master/subfolder/', + }, +}; + +const mockEntityWithAnnotation: Entity = { + ...entityBase, + ...{ + metadata: { + ...metadataBase, + ...goodAnnotation, + }, + }, +}; + +const badAnnotation = { + annotations: { + 'backstage.io/techdocs-ref': 'bad-annotation', + }, +}; + +const mockEntityWithBadAnnotation: Entity = { + ...entityBase, + ...{ + metadata: { + ...metadataBase, + ...badAnnotation, + }, + }, +}; + +describe('parseReferenceAnnotation', () => { + it('should parse annotation', () => { + const parsedLocationAnnotation = parseReferenceAnnotation( + 'backstage.io/techdocs-ref', + mockEntityWithAnnotation, + ); + expect(parsedLocationAnnotation.type).toBe('url'); + expect(parsedLocationAnnotation.target).toBe( + 'https://github.com/backstage/backstage/blob/master/subfolder/', + ); + }); + + it('should throw error without annotation', () => { + expect(() => { + parseReferenceAnnotation('backstage.io/techdocs-ref', entityBase); + }).toThrow(/No location annotation/); + }); + + it('should throw error with bad annotation', () => { + expect(() => { + parseReferenceAnnotation( + 'backstage.io/techdocs-ref', + mockEntityWithBadAnnotation, + ); + }).toThrow(/Failure to parse/); + }); +}); + +describe('getLocationForEntity', () => { + it('should get location for entity', () => { + const parsedLocationAnnotation = getLocationForEntity( + mockEntityWithAnnotation, + ); + expect(parsedLocationAnnotation.type).toBe('url'); + expect(parsedLocationAnnotation.target).toBe( + 'https://github.com/backstage/backstage/blob/master/subfolder/', + ); + }); +}); + +describe('getDocFilesFromRepository', () => { + it('should read a remote directory using UrlReader.readTree', async () => { + class MockUrlReader implements UrlReader { + async read() { + return Buffer.from('mock'); + } + + async readTree(): Promise { + return { + dir: async () => { + return '/tmp/testfolder'; + }, + files: async () => { + return []; + }, + archive: async () => { + return Readable.from(''); + }, + }; + } + } + + const output = await getDocFilesFromRepository( + new MockUrlReader(), + mockEntityWithAnnotation, + ); + + expect(output).toBe('/tmp/testfolder'); + }); +}); diff --git a/plugins/techdocs-backend/src/helpers.ts b/packages/techdocs-common/src/helpers.ts similarity index 98% rename from plugins/techdocs-backend/src/helpers.ts rename to packages/techdocs-common/src/helpers.ts index fcf5a4f194..e08d58d58e 100644 --- a/plugins/techdocs-backend/src/helpers.ts +++ b/packages/techdocs-common/src/helpers.ts @@ -23,7 +23,7 @@ import { getDefaultBranch } from './default-branch'; import { getGitRepoType, getTokenForGitRepo } from './git-auth'; import { Entity } from '@backstage/catalog-model'; import { InputError, UrlReader } from '@backstage/backend-common'; -import { RemoteProtocol } from './techdocs/stages/prepare/types'; +import { RemoteProtocol } from './stages/prepare/types'; import { Logger } from 'winston'; // Enables core.longpaths on windows to prevent crashing when checking out repos with long foldernames and/or deep nesting diff --git a/plugins/techdocs-backend/src/techdocs/index.ts b/packages/techdocs-common/src/index.ts similarity index 87% rename from plugins/techdocs-backend/src/techdocs/index.ts rename to packages/techdocs-common/src/index.ts index 7113525bb8..a6e1831049 100644 --- a/plugins/techdocs-backend/src/techdocs/index.ts +++ b/packages/techdocs-common/src/index.ts @@ -14,3 +14,6 @@ * limitations under the License. */ export * from './stages'; +export * from './helpers'; +export * from './default-branch'; +export * from './git-auth'; diff --git a/plugins/techdocs-backend/src/techdocs/stages/generate/__fixtures__/mkdocs.yml b/packages/techdocs-common/src/stages/generate/__fixtures__/mkdocs.yml similarity index 100% rename from plugins/techdocs-backend/src/techdocs/stages/generate/__fixtures__/mkdocs.yml rename to packages/techdocs-common/src/stages/generate/__fixtures__/mkdocs.yml diff --git a/plugins/techdocs-backend/src/techdocs/stages/generate/__fixtures__/mkdocs_with_repo_url.yml b/packages/techdocs-common/src/stages/generate/__fixtures__/mkdocs_with_repo_url.yml similarity index 100% rename from plugins/techdocs-backend/src/techdocs/stages/generate/__fixtures__/mkdocs_with_repo_url.yml rename to packages/techdocs-common/src/stages/generate/__fixtures__/mkdocs_with_repo_url.yml diff --git a/plugins/techdocs-backend/src/techdocs/stages/generate/generators.test.ts b/packages/techdocs-common/src/stages/generate/generators.test.ts similarity index 96% rename from plugins/techdocs-backend/src/techdocs/stages/generate/generators.test.ts rename to packages/techdocs-common/src/stages/generate/generators.test.ts index a9303c3794..7f4a964ec7 100644 --- a/plugins/techdocs-backend/src/techdocs/stages/generate/generators.test.ts +++ b/packages/techdocs-common/src/stages/generate/generators.test.ts @@ -14,7 +14,7 @@ * limitations under the License. */ -import { Generators, TechdocsGenerator } from './'; +import { Generators, TechdocsGenerator } from '.'; import { getVoidLogger } from '@backstage/backend-common'; import { ConfigReader } from '@backstage/config'; diff --git a/plugins/techdocs-backend/src/techdocs/stages/generate/generators.ts b/packages/techdocs-common/src/stages/generate/generators.ts similarity index 100% rename from plugins/techdocs-backend/src/techdocs/stages/generate/generators.ts rename to packages/techdocs-common/src/stages/generate/generators.ts diff --git a/plugins/techdocs-backend/src/techdocs/stages/generate/helpers.test.ts b/packages/techdocs-common/src/stages/generate/helpers.test.ts similarity index 99% rename from plugins/techdocs-backend/src/techdocs/stages/generate/helpers.test.ts rename to packages/techdocs-common/src/stages/generate/helpers.test.ts index 54ef0b1e83..83ccbbffab 100644 --- a/plugins/techdocs-backend/src/techdocs/stages/generate/helpers.test.ts +++ b/packages/techdocs-common/src/stages/generate/helpers.test.ts @@ -28,7 +28,7 @@ import { patchMkdocsYmlPreBuild, } from './helpers'; import { RemoteProtocol } from '../prepare/types'; -import { ParsedLocationAnnotation } from '../../../helpers'; +import { ParsedLocationAnnotation } from '../../helpers'; const mockEntity = { apiVersion: 'version', diff --git a/plugins/techdocs-backend/src/techdocs/stages/generate/helpers.ts b/packages/techdocs-common/src/stages/generate/helpers.ts similarity index 99% rename from plugins/techdocs-backend/src/techdocs/stages/generate/helpers.ts rename to packages/techdocs-common/src/stages/generate/helpers.ts index 0f60712f5e..773b542517 100644 --- a/plugins/techdocs-backend/src/techdocs/stages/generate/helpers.ts +++ b/packages/techdocs-common/src/stages/generate/helpers.ts @@ -22,7 +22,7 @@ import yaml from 'js-yaml'; import { Logger } from 'winston'; import { Entity } from '@backstage/catalog-model'; import { SupportedGeneratorKey } from './types'; -import { ParsedLocationAnnotation } from '../../../helpers'; +import { ParsedLocationAnnotation } from '../../helpers'; import { RemoteProtocol } from '../prepare/types'; // TODO: Implement proper support for more generators. @@ -209,7 +209,7 @@ export const getRepoUrlFromLocationAnnotation = ( }; /** - * Update the mkdocs.yml file before TechDocs generator uses it to build docs site. + * Update the mkdocs.yml file before TechDocs generator uses it to generate docs site. * * List of tasks: * - Add repo_url if it does not exists diff --git a/plugins/techdocs-backend/src/techdocs/stages/generate/index.ts b/packages/techdocs-common/src/stages/generate/index.ts similarity index 100% rename from plugins/techdocs-backend/src/techdocs/stages/generate/index.ts rename to packages/techdocs-common/src/stages/generate/index.ts diff --git a/plugins/techdocs-backend/src/techdocs/stages/generate/techdocs.ts b/packages/techdocs-common/src/stages/generate/techdocs.ts similarity index 100% rename from plugins/techdocs-backend/src/techdocs/stages/generate/techdocs.ts rename to packages/techdocs-common/src/stages/generate/techdocs.ts diff --git a/plugins/techdocs-backend/src/techdocs/stages/generate/types.ts b/packages/techdocs-common/src/stages/generate/types.ts similarity index 97% rename from plugins/techdocs-backend/src/techdocs/stages/generate/types.ts rename to packages/techdocs-common/src/stages/generate/types.ts index 6d9ce5afca..65e411c3aa 100644 --- a/plugins/techdocs-backend/src/techdocs/stages/generate/types.ts +++ b/packages/techdocs-common/src/stages/generate/types.ts @@ -16,7 +16,7 @@ import { Writable } from 'stream'; import Docker from 'dockerode'; import { Entity } from '@backstage/catalog-model'; -import { ParsedLocationAnnotation } from '../../../helpers'; +import { ParsedLocationAnnotation } from '../../helpers'; /** * The returned directory from the generator which is ready diff --git a/plugins/techdocs-backend/src/techdocs/stages/index.ts b/packages/techdocs-common/src/stages/index.ts similarity index 100% rename from plugins/techdocs-backend/src/techdocs/stages/index.ts rename to packages/techdocs-common/src/stages/index.ts diff --git a/plugins/techdocs-backend/src/techdocs/stages/prepare/commonGit.test.ts b/packages/techdocs-common/src/stages/prepare/commonGit.test.ts similarity index 95% rename from plugins/techdocs-backend/src/techdocs/stages/prepare/commonGit.test.ts rename to packages/techdocs-common/src/stages/prepare/commonGit.test.ts index 843df476e9..a4240514b5 100644 --- a/plugins/techdocs-backend/src/techdocs/stages/prepare/commonGit.test.ts +++ b/packages/techdocs-common/src/stages/prepare/commonGit.test.ts @@ -16,7 +16,7 @@ import { getVoidLogger } from '@backstage/backend-common'; import { CommonGitPreparer } from './commonGit'; -import { checkoutGitRepository } from '../../../helpers'; +import { checkoutGitRepository } from '../../helpers'; function normalizePath(path: string) { return path @@ -25,8 +25,8 @@ function normalizePath(path: string) { .join('/'); } -jest.mock('../../../helpers', () => ({ - ...jest.requireActual<{}>('../../../helpers'), +jest.mock('../../helpers', () => ({ + ...jest.requireActual<{}>('../../helpers'), checkoutGitRepository: jest.fn(() => '/tmp/backstage-repo/org/name/branch'), })); diff --git a/plugins/techdocs-backend/src/techdocs/stages/prepare/commonGit.ts b/packages/techdocs-common/src/stages/prepare/commonGit.ts similarity index 94% rename from plugins/techdocs-backend/src/techdocs/stages/prepare/commonGit.ts rename to packages/techdocs-common/src/stages/prepare/commonGit.ts index d9ba96a031..d79373fba3 100644 --- a/plugins/techdocs-backend/src/techdocs/stages/prepare/commonGit.ts +++ b/packages/techdocs-common/src/stages/prepare/commonGit.ts @@ -17,10 +17,7 @@ import path from 'path'; import { Entity } from '@backstage/catalog-model'; import { PreparerBase } from './types'; import parseGitUrl from 'git-url-parse'; -import { - parseReferenceAnnotation, - checkoutGitRepository, -} from '../../../helpers'; +import { parseReferenceAnnotation, checkoutGitRepository } from '../../helpers'; import { Logger } from 'winston'; diff --git a/plugins/techdocs-backend/src/techdocs/stages/prepare/dir.test.ts b/packages/techdocs-common/src/stages/prepare/dir.test.ts similarity index 95% rename from plugins/techdocs-backend/src/techdocs/stages/prepare/dir.test.ts rename to packages/techdocs-common/src/stages/prepare/dir.test.ts index dc2b1d7d48..5b51f1f46d 100644 --- a/plugins/techdocs-backend/src/techdocs/stages/prepare/dir.test.ts +++ b/packages/techdocs-common/src/stages/prepare/dir.test.ts @@ -15,7 +15,7 @@ */ import { DirectoryPreparer } from './dir'; import { getVoidLogger } from '@backstage/backend-common'; -import { checkoutGitRepository } from '../../../helpers'; +import { checkoutGitRepository } from '../../helpers'; function normalizePath(path: string) { return path @@ -24,8 +24,8 @@ function normalizePath(path: string) { .join('/'); } -jest.mock('../../../helpers', () => ({ - ...jest.requireActual<{}>('../../../helpers'), +jest.mock('../../helpers', () => ({ + ...jest.requireActual<{}>('../../helpers'), checkoutGitRepository: jest.fn(() => '/tmp/backstage-repo/org/name/branch/'), })); diff --git a/plugins/techdocs-backend/src/techdocs/stages/prepare/dir.ts b/packages/techdocs-common/src/stages/prepare/dir.ts similarity index 96% rename from plugins/techdocs-backend/src/techdocs/stages/prepare/dir.ts rename to packages/techdocs-common/src/stages/prepare/dir.ts index 8faca33606..3485ecb5dd 100644 --- a/plugins/techdocs-backend/src/techdocs/stages/prepare/dir.ts +++ b/packages/techdocs-common/src/stages/prepare/dir.ts @@ -16,10 +16,7 @@ import { PreparerBase } from './types'; import { Entity } from '@backstage/catalog-model'; import path from 'path'; -import { - parseReferenceAnnotation, - checkoutGitRepository, -} from '../../../helpers'; +import { parseReferenceAnnotation, checkoutGitRepository } from '../../helpers'; import { InputError } from '@backstage/backend-common'; import parseGitUrl from 'git-url-parse'; import { Logger } from 'winston'; diff --git a/plugins/techdocs-backend/src/techdocs/stages/prepare/index.ts b/packages/techdocs-common/src/stages/prepare/index.ts similarity index 100% rename from plugins/techdocs-backend/src/techdocs/stages/prepare/index.ts rename to packages/techdocs-common/src/stages/prepare/index.ts diff --git a/plugins/techdocs-backend/src/techdocs/stages/prepare/preparers.ts b/packages/techdocs-common/src/stages/prepare/preparers.ts similarity index 95% rename from plugins/techdocs-backend/src/techdocs/stages/prepare/preparers.ts rename to packages/techdocs-common/src/stages/prepare/preparers.ts index 2f0df47de4..52a47957e8 100644 --- a/plugins/techdocs-backend/src/techdocs/stages/prepare/preparers.ts +++ b/packages/techdocs-common/src/stages/prepare/preparers.ts @@ -16,7 +16,7 @@ import { PreparerBase, RemoteProtocol, PreparerBuilder } from './types'; import { Entity } from '@backstage/catalog-model'; -import { parseReferenceAnnotation } from '../../../helpers'; +import { parseReferenceAnnotation } from '../../helpers'; export class Preparers implements PreparerBuilder { private preparerMap = new Map(); diff --git a/plugins/techdocs-backend/src/techdocs/stages/prepare/types.ts b/packages/techdocs-common/src/stages/prepare/types.ts similarity index 100% rename from plugins/techdocs-backend/src/techdocs/stages/prepare/types.ts rename to packages/techdocs-common/src/stages/prepare/types.ts diff --git a/plugins/techdocs-backend/src/techdocs/stages/prepare/url.ts b/packages/techdocs-common/src/stages/prepare/url.ts similarity index 95% rename from plugins/techdocs-backend/src/techdocs/stages/prepare/url.ts rename to packages/techdocs-common/src/stages/prepare/url.ts index 330db05aa4..3407684837 100644 --- a/plugins/techdocs-backend/src/techdocs/stages/prepare/url.ts +++ b/packages/techdocs-common/src/stages/prepare/url.ts @@ -15,8 +15,7 @@ */ import { Entity } from '@backstage/catalog-model'; import { PreparerBase } from './types'; -import { getDocFilesFromRepository } from '../../../helpers'; - +import { getDocFilesFromRepository } from '../../helpers'; import { Logger } from 'winston'; import { UrlReader } from '@backstage/backend-common'; diff --git a/packages/techdocs-common/src/stages/publish/googleStorage.test.ts b/packages/techdocs-common/src/stages/publish/googleStorage.test.ts new file mode 100644 index 0000000000..2268ea3fcc --- /dev/null +++ b/packages/techdocs-common/src/stages/publish/googleStorage.test.ts @@ -0,0 +1,84 @@ +/* + * Copyright 2020 Spotify AB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import mockFs from 'mock-fs'; +import * as winston from 'winston'; +import { ConfigReader } from '@backstage/config'; +import { GoogleGCSPublish } from './googleStorage'; +import { PublisherBase } from './types'; + +const createMockEntity = (annotations = {}) => { + return { + apiVersion: 'version', + kind: 'TestKind', + metadata: { + name: 'test-component-name', + annotations: { + ...annotations, + }, + }, + }; +}; + +const logger = winston.createLogger(); +jest.spyOn(logger, 'info').mockReturnValue(logger); + +let publisher: PublisherBase; + +beforeEach(() => { + const mockConfig = ConfigReader.fromConfigs([ + { + context: '', + data: { + techdocs: { + requestUrl: 'http://localhost:7000', + publisher: { + type: 'googleGcs', + googleGcs: { + credentials: '{}', + projectId: 'gcp-project-id', + bucketName: 'bucketName', + }, + }, + }, + }, + }, + ]); + + publisher = GoogleGCSPublish.fromConfig(mockConfig, logger); +}); + +describe('GoogleGCSPublish', () => { + it('should publish a directory', async () => { + mockFs({ + '/path/to/generatedDirectory': { + 'index.html': '', + '404.html': '', + assets: { + 'main.css': '', + }, + }, + }); + + const entity = createMockEntity(); + expect( + await publisher.publish({ + entity, + directory: '/path/to/generatedDirectory', + }), + ).toBeUndefined(); + mockFs.restore(); + }); +}); diff --git a/packages/techdocs-common/src/stages/publish/googleStorage.ts b/packages/techdocs-common/src/stages/publish/googleStorage.ts new file mode 100644 index 0000000000..5f8dc6a3fc --- /dev/null +++ b/packages/techdocs-common/src/stages/publish/googleStorage.ts @@ -0,0 +1,218 @@ +/* + * Copyright 2020 Spotify AB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import express from 'express'; +import { + Storage, + UploadResponse, + FileExistsResponse, +} from '@google-cloud/storage'; +import { Logger } from 'winston'; +import { Entity, EntityName } from '@backstage/catalog-model'; +import { Config } from '@backstage/config'; +import { getHeadersForFileExtension, getFileTreeRecursively } from './helpers'; +import { PublisherBase, PublishRequest } from './types'; + +export class GoogleGCSPublish implements PublisherBase { + static fromConfig(config: Config, logger: Logger): PublisherBase { + let credentials = ''; + let projectId = ''; + let bucketName = ''; + try { + credentials = config.getString( + 'techdocs.publisher.googleGcs.credentials', + ); + projectId = config.getString('techdocs.publisher.googleGcs.projectId'); + bucketName = config.getString('techdocs.publisher.googleGcs.bucketName'); + } catch (error) { + throw new Error( + "Since techdocs.publisher.type is set to 'googleGcs' in your app config, " + + 'credentials, projectId and bucketName are required in techdocs.publisher.googleGcs ' + + 'required to authenticate with Google Cloud Storage.', + ); + } + + let credentialsJson = {}; + try { + credentialsJson = JSON.parse(credentials); + } catch (err) { + throw new Error( + 'Error in parsing techdocs.publisher.googleGcs.credentials config to JSON.', + ); + } + + const storageClient = new Storage({ + credentials: credentialsJson, + projectId: projectId, + }); + + // Check if the defined bucket exists. Being able to connect means the configuration is good + // and the storage client will work. + storageClient + .bucket(bucketName) + .getMetadata() + .then(() => { + logger.info( + `Successfully connected to the GCS bucket ${bucketName} in the GCP project ${projectId}.`, + ); + }) + .catch(reason => { + logger.error( + `Could not retrieve metadata about the GCS bucket ${bucketName} in the GCP project ${projectId}. ` + + 'Make sure the GCP project and the bucket exists and the access key located at the path ' + + "techdocs.publisher.googleGcs.credentials defined in app config has the role 'Storage Object Creator'. " + + 'Refer to https://backstage.io/docs/features/techdocs/using-cloud-storage', + ); + throw new Error(`from GCS client library: ${reason.message}`); + }); + + return new GoogleGCSPublish(storageClient, bucketName, logger); + } + + constructor( + private readonly storageClient: Storage, + private readonly bucketName: string, + private readonly logger: Logger, + ) { + this.storageClient = storageClient; + this.bucketName = bucketName; + this.logger = logger; + } + + /** + * Upload all the files from the generated `directory` to the GCS bucket. + * Directory structure used in the bucket is - entityNamespace/entityKind/entityName/index.html + */ + publish({ entity, directory }: PublishRequest): Promise { + return new Promise(async (resolve, reject) => { + // Note: GCS manages creation of parent directories if they do not exist. + // So collecting path of only the files is good enough. + const allFilesToUpload = await getFileTreeRecursively(directory); + + const uploadPromises: Array> = []; + allFilesToUpload.forEach(filePath => { + // Remove the absolute path prefix of the source directory + // Path of all files to upload, relative to the root of the source directory + // e.g. ['index.html', 'sub-page/index.html', 'assets/images/favicon.png'] + const relativeFilePath = filePath.replace(`${directory}/`, ''); + const entityRootDir = `${entity.metadata.namespace}/${entity.kind}/${entity.metadata.name}`; + const destination = `${entityRootDir}/${relativeFilePath}`; // GCS Bucket file relative path + // TODO: Upload in chunks of ~10 files instead of all files at once. + uploadPromises.push( + this.storageClient.bucket(this.bucketName).upload(filePath, { + destination, + }), + ); + }); + + Promise.all(uploadPromises) + .then(() => { + this.logger.info( + `Successfully uploaded all the generated files for Entity ${entity.metadata.name}. Total number of files: ${allFilesToUpload.length}`, + ); + resolve(undefined); + }) + .catch((err: Error) => { + const errorMessage = `Unable to upload file(s) to Google Cloud Storage. Error ${err.message}`; + this.logger.error(errorMessage); + reject(errorMessage); + }); + }); + } + + fetchTechDocsMetadata(entityName: EntityName): Promise { + return new Promise((resolve, reject) => { + const entityRootDir = `${entityName.namespace}/${entityName.kind}/${entityName.name}`; + + const fileStreamChunks: Array = []; + this.storageClient + .bucket(this.bucketName) + .file(`${entityRootDir}/techdocs_metadata.json`) + .createReadStream() + .on('error', err => { + this.logger.error(err.message); + reject(err.message); + }) + .on('data', chunk => { + fileStreamChunks.push(chunk); + }) + .on('end', () => { + const techdocsMetadataJson = Buffer.concat( + fileStreamChunks, + ).toString(); + resolve(techdocsMetadataJson); + }); + }); + } + + /** + * Express route middleware to serve static files on a route in techdocs-backend. + */ + docsRouter(): express.Handler { + return (req, res) => { + // Trim the leading forward slash + // filePath example - /default/Component/documented-component/index.html + const filePath = req.path.replace(/^\//, ''); + + // Files with different extensions (CSS, HTML) need to be served with different headers + const fileExtension = filePath.split('.')[filePath.split('.').length - 1]; + const responseHeaders = getHeadersForFileExtension(fileExtension); + + const fileStreamChunks: Array = []; + this.storageClient + .bucket(this.bucketName) + .file(filePath) + .createReadStream() + .on('error', err => { + this.logger.warn(err.message); + res.status(404).send(err.message); + }) + .on('data', chunk => { + fileStreamChunks.push(chunk); + }) + .on('end', () => { + const fileContent = Buffer.concat(fileStreamChunks).toString(); + // Inject response headers + for (const [headerKey, headerValue] of Object.entries( + responseHeaders, + )) { + res.setHeader(headerKey, headerValue); + } + + res.send(fileContent); + }); + }; + } + + /** + * A helper function which checks if index.html of an Entity's docs site is available. This + * can be used to verify if there are any pre-generated docs available to serve. + */ + async hasDocsBeenGenerated(entity: Entity): Promise { + return new Promise(resolve => { + const entityRootDir = `${entity.metadata.namespace}/${entity.kind}/${entity.metadata.name}`; + this.storageClient + .bucket(this.bucketName) + .file(`${entityRootDir}/index.html`) + .exists() + .then((response: FileExistsResponse) => { + resolve(response[0]); + }) + .catch(() => { + resolve(false); + }); + }); + } +} diff --git a/packages/techdocs-common/src/stages/publish/helpers.test.ts b/packages/techdocs-common/src/stages/publish/helpers.test.ts new file mode 100644 index 0000000000..e4f0c12b79 --- /dev/null +++ b/packages/techdocs-common/src/stages/publish/helpers.test.ts @@ -0,0 +1,69 @@ +/* + * Copyright 2020 Spotify AB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import mockFs from 'mock-fs'; +import { getFileTreeRecursively, getHeadersForFileExtension } from './helpers'; + +describe('getHeadersForFileExtension', () => { + it('returns correct header for default extensions', () => { + const headers = getHeadersForFileExtension('xyz'); + const expectedHeaders = { + 'Content-Type': 'text/plain', + }; + expect(headers).toEqual(expectedHeaders); + }); + + it('returns correct header for html', () => { + const headers = getHeadersForFileExtension('html'); + const expectedHeaders = { + 'Content-Type': 'text/html; charset=UTF-8', + }; + expect(headers).toEqual(expectedHeaders); + }); + + it('returns correct header for css', () => { + const headers = getHeadersForFileExtension('css'); + const expectedHeaders = { + 'Content-Type': 'text/css; charset=UTF-8', + }; + expect(headers).toEqual(expectedHeaders); + }); +}); + +describe('getFileTreeRecursively', () => { + beforeEach(() => { + mockFs({ + '/rootDir': { + file1: '', + subDirA: { + file2: '', + emptyDir1: mockFs.directory(), + }, + emptyDir2: mockFs.directory(), + }, + }); + }); + + afterEach(() => { + mockFs.restore(); + }); + + it('returns complete file tree of a path', async () => { + const fileList = await getFileTreeRecursively('/rootDir'); + expect(fileList.length).toBe(2); + expect(fileList).toContain('/rootDir/file1'); + expect(fileList).toContain('/rootDir/subDirA/file2'); + }); +}); diff --git a/packages/techdocs-common/src/stages/publish/helpers.ts b/packages/techdocs-common/src/stages/publish/helpers.ts new file mode 100644 index 0000000000..38ff27e0a0 --- /dev/null +++ b/packages/techdocs-common/src/stages/publish/helpers.ts @@ -0,0 +1,85 @@ +/* + * Copyright 2020 Spotify AB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import recursiveReadDir from 'recursive-readdir'; + +export type responseHeadersType = { + 'Content-Type': string; +}; + +/** + * Some files need special headers to be used correctly by the frontend. This function + * generates headers in the response to those file requests. + * @param {string} fileExtension html, css, js etc. + */ +export const getHeadersForFileExtension = ( + fileExtension: string, +): responseHeadersType => { + const headersCommon = { + 'Content-Type': 'text/plain', + }; + const headersHTML = { + ...headersCommon, + 'Content-Type': 'text/html; charset=UTF-8', + }; + + const headersCSS = { + ...headersCommon, + 'Content-Type': 'text/css; charset=UTF-8', + }; + + switch (fileExtension) { + case 'html': + return headersHTML; + case 'css': + return headersCSS; + default: + return headersCommon; + } +}; + +/** + * Recursively traverse all the sub-directories of a path and return + * a list of absolute paths of all the files. e.g. tree command in Unix + * + * @example + * + * /User/username/my_dir + * dirA + * | subDirA + * | | file1 + * EmptyDir + * dirB + * | file2 + * file3 + * + * getFileListRecursively('/Users/username/myDir') + * // returns + * [ + * '/User/username/my_dir/dirA/subDirA/file1', + * '/User/username/my_dir/dirB/file2', + * '/User/username/my_dir/file3' + * ] + * @param rootDirPath Absolute path to the root directory. + */ +export const getFileTreeRecursively = async ( + rootDirPath: string, +): Promise => { + // Iterate on all the files in the directory and its sub-directories + const fileList = await recursiveReadDir(rootDirPath).catch(error => { + throw new Error(`Failed to read template directory: ${error.message}`); + }); + return fileList; +}; diff --git a/plugins/techdocs-backend/src/techdocs/stages/publish/index.ts b/packages/techdocs-common/src/stages/publish/index.ts similarity index 85% rename from plugins/techdocs-backend/src/techdocs/stages/publish/index.ts rename to packages/techdocs-common/src/stages/publish/index.ts index 0029ea5c4e..494efe5ac1 100644 --- a/plugins/techdocs-backend/src/techdocs/stages/publish/index.ts +++ b/packages/techdocs-common/src/stages/publish/index.ts @@ -13,5 +13,5 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -export { LocalPublish } from './local'; -export type { PublisherBase } from './types'; +export { Publisher } from './publish'; +export type { PublisherBase, PublisherType } from './types'; diff --git a/plugins/techdocs-backend/src/techdocs/stages/publish/local.test.ts b/packages/techdocs-common/src/stages/publish/local.test.ts similarity index 62% rename from plugins/techdocs-backend/src/techdocs/stages/publish/local.test.ts rename to packages/techdocs-common/src/stages/publish/local.test.ts index 18344489e1..665ba28c52 100644 --- a/plugins/techdocs-backend/src/techdocs/stages/publish/local.test.ts +++ b/packages/techdocs-common/src/stages/publish/local.test.ts @@ -21,8 +21,26 @@ import { getVoidLogger, PluginEndpointDiscovery, } from '@backstage/backend-common'; +import { ConfigReader } from '@backstage/config'; import { LocalPublish } from './local'; +jest.mock('fs-extra', () => { + const fsOriginal = jest.requireActual('fs-extra'); + return { + ...fsOriginal, + access: jest.fn().mockImplementation((path, checkType, callback) => { + if ( + path.includes('http://localhost:7000/static') && + checkType === fs.constants.F_OK + ) { + callback(); + } else { + callback(new Error()); + } + }), + }; +}); + const createMockEntity = (annotations = {}) => { return { apiVersion: 'version', @@ -41,34 +59,44 @@ const logger = getVoidLogger(); describe('local publisher', () => { it('should publish generated documentation dir', async () => { const testDiscovery: jest.Mocked = { - getBaseUrl: jest.fn().mockResolvedValueOnce('http://localhost:7000'), + getBaseUrl: jest.fn().mockResolvedValue('http://localhost:7000'), getExternalBaseUrl: jest.fn(), }; - const publisher = new LocalPublish(logger, testDiscovery); + const mockConfig = ConfigReader.fromConfigs([ + { + context: '', + data: { + techdocs: { + requestUrl: 'http://localhost:7000', + storageUrl: 'http://localhost:7000/static/docs', + }, + }, + }, + ]); + const publisher = new LocalPublish(mockConfig, logger, testDiscovery); const mockEntity = createMockEntity(); - const tempDir = fs.mkdtempSync(`${__dirname}/test-component-folder-`); - expect(tempDir).toBeTruthy(); fs.closeSync(fs.openSync(path.join(tempDir, '/mock-file'), 'w')); - await publisher.publish({ entity: mockEntity, directory: tempDir }); + const publishDir = path.resolve( __dirname, - `../../../../static/docs/${mockEntity.metadata.name}`, + `../../../../../plugins/techdocs-backend/static/docs/${mockEntity.metadata.name}`, ); - const resultDir = path.resolve( __dirname, - `../../../../static/docs/default/${mockEntity.kind}/${mockEntity.metadata.name}`, + `../../../../../plugins/techdocs-backend/static/docs/default/${mockEntity.kind}/${mockEntity.metadata.name}`, ); expect(fs.existsSync(resultDir)).toBeTruthy(); expect(fs.existsSync(path.join(resultDir, '/mock-file'))).toBeTruthy(); + expect(await publisher.hasDocsBeenGenerated(mockEntity)).toBe(true); + fs.removeSync(publishDir); fs.removeSync(tempDir); }); diff --git a/packages/techdocs-common/src/stages/publish/local.ts b/packages/techdocs-common/src/stages/publish/local.ts new file mode 100644 index 0000000000..2739940530 --- /dev/null +++ b/packages/techdocs-common/src/stages/publish/local.ts @@ -0,0 +1,147 @@ +/* + * Copyright 2020 Spotify AB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import fetch from 'cross-fetch'; +import express from 'express'; +import fs from 'fs-extra'; +import { Logger } from 'winston'; +import { Entity, EntityName } from '@backstage/catalog-model'; +import { + resolvePackagePath, + PluginEndpointDiscovery, +} from '@backstage/backend-common'; +import { Config } from '@backstage/config'; +import { PublisherBase, PublishRequest, PublishResponse } from './types'; + +const staticDocsDir = resolvePackagePath( + '@backstage/plugin-techdocs-backend', + 'static/docs', +); + +/** + * Local publisher which uses the local filesystem to store the generated static files. It uses a directory + * called "static" at the root of techdocs-backend plugin. + */ +export class LocalPublish implements PublisherBase { + private readonly config: Config; + private readonly logger: Logger; + private readonly discovery: PluginEndpointDiscovery; + + constructor( + config: Config, + logger: Logger, + discovery: PluginEndpointDiscovery, + ) { + this.config = config; + this.logger = logger; + this.discovery = discovery; + } + + publish({ entity, directory }: PublishRequest): Promise { + const entityNamespace = entity.metadata.namespace ?? 'default'; + + const publishDir = resolvePackagePath( + '@backstage/plugin-techdocs-backend', + 'static/docs', + entityNamespace, + entity.kind, + entity.metadata.name, + ); + + if (!fs.existsSync(publishDir)) { + this.logger.info(`Could not find ${publishDir}, creating the directory.`); + fs.mkdirSync(publishDir, { recursive: true }); + } + + return new Promise((resolve, reject) => { + fs.copy(directory, publishDir, err => { + if (err) { + this.logger.debug( + `Failed to copy docs from ${directory} to ${publishDir}`, + ); + reject(err); + } + + this.discovery + .getBaseUrl('techdocs') + .then(techdocsApiUrl => { + resolve({ + remoteUrl: `${techdocsApiUrl}/static/docs/${entity.metadata.name}`, + }); + }) + .catch(reason => { + reject(reason); + }); + }); + }); + } + + fetchTechDocsMetadata(entityName: EntityName): Promise { + return new Promise((resolve, reject) => { + this.discovery.getBaseUrl('techdocs').then(techdocsApiUrl => { + const storageUrl = new URL( + new URL(this.config.getString('techdocs.storageUrl')).pathname, + techdocsApiUrl, + ).toString(); + + const entityRootDir = `${entityName.namespace}/${entityName.kind}/${entityName.name}`; + const metadataURL = `${storageUrl}/${entityRootDir}/techdocs_metadata.json`; + fetch(metadataURL) + .then(response => + response + .json() + .then(techdocsMetadataJson => resolve(techdocsMetadataJson)) + .catch(err => { + reject( + `Unable to parse metadata JSON for ${entityRootDir}. Error: ${err}`, + ); + }), + ) + .catch(err => { + reject( + `Unable to fetch metadata for ${entityRootDir}. Error ${err}`, + ); + }); + }); + }); + } + + docsRouter(): express.Handler { + return express.static(staticDocsDir); + } + + async hasDocsBeenGenerated(entity: Entity): Promise { + const namespace = entity.metadata.namespace ?? 'default'; + return new Promise(resolve => { + this.discovery.getBaseUrl('techdocs').then(techdocsApiUrl => { + const storageUrl = new URL( + new URL(this.config.getString('techdocs.storageUrl')).pathname, + techdocsApiUrl, + ).toString(); + + const entityRootDir = `${namespace}/${entity.kind}/${entity.metadata.name}`; + const indexHtmlUrl = `${storageUrl}/${entityRootDir}/index.html`; + // Check if the file exists + fs.access(indexHtmlUrl, fs.constants.F_OK, err => { + if (err) { + resolve(false); + } else { + resolve(true); + } + }); + }); + }); + } +} diff --git a/packages/techdocs-common/src/stages/publish/publish.test.ts b/packages/techdocs-common/src/stages/publish/publish.test.ts new file mode 100644 index 0000000000..244a606129 --- /dev/null +++ b/packages/techdocs-common/src/stages/publish/publish.test.ts @@ -0,0 +1,90 @@ +/* + * Copyright 2020 Spotify AB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import { + getVoidLogger, + PluginEndpointDiscovery, +} from '@backstage/backend-common'; +import { ConfigReader } from '@backstage/config'; +import { Publisher } from './publish'; +import { LocalPublish } from './local'; +import { GoogleGCSPublish } from './googleStorage'; + +const logger = getVoidLogger(); +const testDiscovery: jest.Mocked = { + getBaseUrl: jest.fn().mockResolvedValueOnce('http://localhost:7000'), + getExternalBaseUrl: jest.fn(), +}; + +describe('Publisher', () => { + it('should create local publisher by default', () => { + const mockConfig = ConfigReader.fromConfigs([ + { + context: '', + data: { + techdocs: { + requestUrl: 'http://localhost:7000', + }, + }, + }, + ]); + + const publisher = Publisher.fromConfig(mockConfig, logger, testDiscovery); + expect(publisher).toBeInstanceOf(LocalPublish); + }); + + it('should create local publisher from config', () => { + const mockConfig = ConfigReader.fromConfigs([ + { + context: '', + data: { + techdocs: { + requestUrl: 'http://localhost:7000', + publisher: { + type: 'local', + }, + }, + }, + }, + ]); + + const publisher = Publisher.fromConfig(mockConfig, logger, testDiscovery); + expect(publisher).toBeInstanceOf(LocalPublish); + }); + + it('should create google gcs publisher from config', () => { + const mockConfig = ConfigReader.fromConfigs([ + { + context: '', + data: { + techdocs: { + requestUrl: 'http://localhost:7000', + publisher: { + type: 'googleGcs', + googleGcs: { + credentials: '{}', + projectId: 'gcp-project-id', + bucketName: 'bucketName', + }, + }, + }, + }, + }, + ]); + + const publisher = Publisher.fromConfig(mockConfig, logger, testDiscovery); + expect(publisher).toBeInstanceOf(GoogleGCSPublish); + }); +}); diff --git a/packages/techdocs-common/src/stages/publish/publish.ts b/packages/techdocs-common/src/stages/publish/publish.ts new file mode 100644 index 0000000000..04a9d89996 --- /dev/null +++ b/packages/techdocs-common/src/stages/publish/publish.ts @@ -0,0 +1,50 @@ +/* + * Copyright 2020 Spotify AB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import { Logger } from 'winston'; +import { Config } from '@backstage/config'; +import { PluginEndpointDiscovery } from '@backstage/backend-common'; + +import { PublisherType, PublisherBase } from './types'; +import { LocalPublish } from './local'; +import { GoogleGCSPublish } from './googleStorage'; + +/** + * Factory class to create a TechDocs publisher based on defined publisher type in app config. + * Uses `techdocs.publisher.type`. + */ +export class Publisher { + static fromConfig( + config: Config, + logger: Logger, + discovery: PluginEndpointDiscovery, + ): PublisherBase { + const publisherType = (config.getOptionalString( + 'techdocs.publisher.type', + ) ?? 'local') as PublisherType; + + switch (publisherType) { + case 'googleGcs': + logger.info('Creating Google Storage Bucket publisher for TechDocs'); + return GoogleGCSPublish.fromConfig(config, logger); + case 'local': + logger.info('Creating Local publisher for TechDocs'); + return new LocalPublish(config, logger, discovery); + default: + logger.info('Creating Local publisher for TechDocs'); + return new LocalPublish(config, logger, discovery); + } + } +} diff --git a/packages/techdocs-common/src/stages/publish/types.ts b/packages/techdocs-common/src/stages/publish/types.ts new file mode 100644 index 0000000000..9bfd8cb334 --- /dev/null +++ b/packages/techdocs-common/src/stages/publish/types.ts @@ -0,0 +1,64 @@ +/* + * Copyright 2020 Spotify AB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import { Entity, EntityName } from '@backstage/catalog-model'; +import express from 'express'; + +/** + * Key for all the different types of TechDocs publishers that are supported. + */ +export type PublisherType = 'local' | 'googleGcs'; + +export type PublishRequest = { + entity: Entity; + /* The Path to the directory where the generated files are stored. */ + directory: string; +}; + +/* `remoteUrl` is the URL which serves files from the local publisher's static directory. */ +export type PublishResponse = { + remoteUrl?: string; +} | void; + +/** + * Base class for a TechDocs publisher (e.g. Local, Google GCS Bucket, AWS S3, etc.) + * The publisher handles publishing of the generated static files after the prepare and generate steps of TechDocs. + * It also provides APIs to communicate with the storage service. + */ +export interface PublisherBase { + /** + * Store the generated static files onto a storage service (either local filesystem or external service). + * + * @param request Object containing the entity from the service + * catalog, and the directory that contains the generated static files from TechDocs. + */ + publish(request: PublishRequest): Promise; + + /** + * Retrieve TechDocs Metadata about a site e.g. name, contributors, last updated, etc. + * This API uses the techdocs_metadata.json file that co-exists along with the generated docs. + */ + fetchTechDocsMetadata(entityName: EntityName): Promise; + + /** + * Route middleware to serve static documentation files for an entity. + */ + docsRouter(): express.Handler; + + /** + * Check if the index.html is present for the Entity at the Storage location. + */ + hasDocsBeenGenerated(entityName: Entity): Promise; +} diff --git a/plugins/techdocs-backend/README.md b/plugins/techdocs-backend/README.md index 625f0910b5..c2a1cb99aa 100644 --- a/plugins/techdocs-backend/README.md +++ b/plugins/techdocs-backend/README.md @@ -18,30 +18,13 @@ yarn start ## What techdocs-backend does -This plugin is the backend part of the techdocs plugin. It provides building and serving of your docs without having to use another service and hosting provider. To use it set your techdocs storageUrl in your `app-config.yml` to `http://localhost:7000/api/techdocs/static/docs`. +This plugin is the backend part of the techdocs plugin. It provides serving and building of documentation for any entity. +To configure various storage providers and building options, see http://backstage.io/docs/features/techdocs/configuration -```yaml -techdocs: - storageUrl: http://localhost:7000/api/techdocs/static/docs -``` - -## Extending techdocs-backend - -Currently the build process of techdocs-backend is split up in these three stages. - -- Preparers -- Generators -- Publishers - -Preparers read your entity data and creates a working directory with your documentation source code. For example if you have set your `backstage.io/techdocs-ref` to `github:https://github.com/backstage/backstage.git` it will clone that repository to a temp folder and pass that on to the generator. - -Generators takes the prepared source and runs the `techdocs-container` on it. It then passes on the output folder of that build to the publisher. - -Publishers gets a folder path from the generator and publish it to your storage solution. Currently the only built in storage solution is a folder called `static/docs` inside the techdocs-backend plugin. - -Any of these can be extended. If we want to publish to a external static file server using rsync for example that can be done by creating a rsync publisher. _(Keep in mind that if you want techdocs-backend to initiate a build this would also require techdocs-backend to act as a proxy, which is not yet implemented.)_ +The techdocs-backend re-exports the [techdocs-common](https://github.com/backstage/backstage/tree/master/packages/techdocs-common) package which has the features to prepare, generate and publish docs. +The Publishers are also used to fetch the static documentation files and render them in TechDocs. ## Links - [Frontend part of the plugin](https://github.com/backstage/backstage/tree/master/plugins/techdocs) -- [The Backstage homepage](https://backstage.io) +- [Backstage homepage](https://backstage.io) diff --git a/plugins/techdocs-backend/package.json b/plugins/techdocs-backend/package.json index 1f8ad9cdab..fc11122ead 100644 --- a/plugins/techdocs-backend/package.json +++ b/plugins/techdocs-backend/package.json @@ -33,19 +33,14 @@ "@backstage/backend-common": "^0.4.0", "@backstage/catalog-model": "^0.5.0", "@backstage/config": "^0.1.2", + "@backstage/techdocs-common": "^0.1.1", "@types/dockerode": "^3.2.1", "@types/express": "^4.17.6", - "command-exists-promise": "^2.0.2", "cross-fetch": "^3.0.6", "dockerode": "^3.2.1", "express": "^4.17.1", "express-promise-router": "^3.0.3", - "fs-extra": "^9.0.1", - "git-url-parse": "^11.4.0", - "js-yaml": "^3.14.0", "knex": "^0.21.6", - "mock-fs": "^4.13.0", - "nodegit": "^0.27.0", "winston": "^3.2.1" }, "devDependencies": { diff --git a/plugins/techdocs-backend/src/techdocs/stages/publish/types.ts b/plugins/techdocs-backend/src/DocsBuilder/BuildMetadataStorage.test.ts similarity index 56% rename from plugins/techdocs-backend/src/techdocs/stages/publish/types.ts rename to plugins/techdocs-backend/src/DocsBuilder/BuildMetadataStorage.test.ts index ca85d9f56e..87de9d1d02 100644 --- a/plugins/techdocs-backend/src/techdocs/stages/publish/types.ts +++ b/plugins/techdocs-backend/src/DocsBuilder/BuildMetadataStorage.test.ts @@ -13,20 +13,15 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -import { Entity } from '@backstage/catalog-model'; +import { BuildMetadataStorage } from './BuildMetadataStorage'; -/** - * Publisher is in charge of taking a folder created by - * the builder, and pushing it to storage - */ -export type PublisherBase = { - /** - * - * @param opts object containing the entity from the service - * catalog, and the directory that has been generated - */ - publish(opts: { - entity: Entity; - directory: string; - }): Promise<{ remoteUrl: string }> | { remoteUrl: string }; -}; +describe('BuildMetadataStorage', () => { + it('should return build timestamp', () => { + const newMetadataStorage = new BuildMetadataStorage('123abc'); + newMetadataStorage.storeBuildTimestamp(); + + const timestamp = newMetadataStorage.getTimestamp(); + + expect(timestamp).toBeLessThanOrEqual(Date.now()); + }); +}); diff --git a/plugins/techdocs-backend/src/storage/BuildMetadataStorage.ts b/plugins/techdocs-backend/src/DocsBuilder/BuildMetadataStorage.ts similarity index 83% rename from plugins/techdocs-backend/src/storage/BuildMetadataStorage.ts rename to plugins/techdocs-backend/src/DocsBuilder/BuildMetadataStorage.ts index b82ec1291a..d6a19764d9 100644 --- a/plugins/techdocs-backend/src/storage/BuildMetadataStorage.ts +++ b/plugins/techdocs-backend/src/DocsBuilder/BuildMetadataStorage.ts @@ -20,6 +20,11 @@ type buildInfo = { const builds = {} as buildInfo; +/** + * Store timestamps of the most recent TechDocs build of each Entity. This is + * used to invalidate cache if the latest commit in the documentation source + * repository is later than the timestamp. + */ export class BuildMetadataStorage { public entityUid: string; private builds: buildInfo; diff --git a/plugins/techdocs-backend/src/DocsBuilder/builder.ts b/plugins/techdocs-backend/src/DocsBuilder/builder.ts new file mode 100644 index 0000000000..67d0dfb89e --- /dev/null +++ b/plugins/techdocs-backend/src/DocsBuilder/builder.ts @@ -0,0 +1,129 @@ +/* + * Copyright 2020 Spotify AB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import Docker from 'dockerode'; +import { Logger } from 'winston'; +import { Entity } from '@backstage/catalog-model'; +import { + PreparerBuilder, + PublisherBase, + GeneratorBuilder, + PreparerBase, + GeneratorBase, + getLocationForEntity, + getLastCommitTimestamp, +} from '@backstage/techdocs-common'; +import { BuildMetadataStorage } from '.'; + +const getEntityId = (entity: Entity) => { + return `${entity.kind}:${entity.metadata.namespace ?? ''}:${ + entity.metadata.name + }`; +}; + +type DocsBuilderArguments = { + preparers: PreparerBuilder; + generators: GeneratorBuilder; + publisher: PublisherBase; + entity: Entity; + logger: Logger; + dockerClient: Docker; +}; + +export class DocsBuilder { + private preparer: PreparerBase; + private generator: GeneratorBase; + private publisher: PublisherBase; + private entity: Entity; + private logger: Logger; + private dockerClient: Docker; + + constructor({ + preparers, + generators, + publisher, + entity, + logger, + dockerClient, + }: DocsBuilderArguments) { + this.preparer = preparers.get(entity); + this.generator = generators.get(entity); + this.publisher = publisher; + this.entity = entity; + this.logger = logger; + this.dockerClient = dockerClient; + } + + public async build() { + this.logger.info(`Running preparer on entity ${getEntityId(this.entity)}`); + const preparedDir = await this.preparer.prepare(this.entity); + + const parsedLocationAnnotation = getLocationForEntity(this.entity); + + this.logger.info(`Running generator on entity ${getEntityId(this.entity)}`); + const { resultDir } = await this.generator.run({ + directory: preparedDir, + dockerClient: this.dockerClient, + parsedLocationAnnotation, + }); + + this.logger.info(`Running publisher on entity ${getEntityId(this.entity)}`); + await this.publisher.publish({ + entity: this.entity, + directory: resultDir, + }); + + if (!this.entity.metadata.uid) { + throw new Error( + 'Trying to build documentation for entity not in service catalog', + ); + } + + new BuildMetadataStorage(this.entity.metadata.uid).storeBuildTimestamp(); + } + + public async docsUpToDate() { + if (!this.entity.metadata.uid) { + throw new Error( + 'Trying to build documentation for entity not in service catalog', + ); + } + + const buildMetadataStorage = new BuildMetadataStorage( + this.entity.metadata.uid, + ); + const { type, target } = getLocationForEntity(this.entity); + + // Unless docs are stored locally + const nonAgeCheckTypes = ['dir', 'file', 'url']; + if (!nonAgeCheckTypes.includes(type)) { + const lastCommit = await getLastCommitTimestamp(target, this.logger); + const storageTimeStamp = buildMetadataStorage.getTimestamp(); + + // Check if documentation source is newer than what we have + if (storageTimeStamp && storageTimeStamp >= lastCommit) { + this.logger.debug( + `Docs for entity ${getEntityId(this.entity)} is up to date.`, + ); + return true; + } + } + + this.logger.debug( + `Docs for entity ${getEntityId(this.entity)} was outdated.`, + ); + return false; + } +} diff --git a/plugins/techdocs-backend/src/storage/index.ts b/plugins/techdocs-backend/src/DocsBuilder/index.ts similarity index 95% rename from plugins/techdocs-backend/src/storage/index.ts rename to plugins/techdocs-backend/src/DocsBuilder/index.ts index 3d37f69679..1380e24e7c 100644 --- a/plugins/techdocs-backend/src/storage/index.ts +++ b/plugins/techdocs-backend/src/DocsBuilder/index.ts @@ -14,3 +14,4 @@ * limitations under the License. */ export * from './BuildMetadataStorage'; +export * from './builder'; diff --git a/plugins/techdocs-backend/src/helpers.test.ts b/plugins/techdocs-backend/src/helpers.test.ts deleted file mode 100644 index 10518df74f..0000000000 --- a/plugins/techdocs-backend/src/helpers.test.ts +++ /dev/null @@ -1,70 +0,0 @@ -/* - * Copyright 2020 Spotify AB - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -import { Readable } from 'stream'; -import { getDocFilesFromRepository } from './helpers'; -import { UrlReader, ReadTreeResponse } from '@backstage/backend-common'; -import { Entity } from '@backstage/catalog-model'; - -describe('getDocFilesFromRepository', () => { - it('should read a remote directory using UrlReader.readTree', async () => { - class MockUrlReader implements UrlReader { - async read() { - return Buffer.from('mock'); - } - - async readTree(): Promise { - return { - dir: async () => { - return '/tmp/testfolder'; - }, - files: async () => { - return []; - }, - archive: async () => { - return Readable.from(''); - }, - }; - } - } - - const mockEntity: Entity = { - metadata: { - namespace: 'default', - annotations: { - 'backstage.io/techdocs-ref': - 'url:https://github.com/backstage/backstage/blob/master/subfolder/', - }, - name: 'mytestcomponent', - description: 'A component for testing', - }, - apiVersion: 'backstage.io/v1alpha1', - kind: 'Component', - spec: { - type: 'documentation', - lifecycle: 'experimental', - owner: 'testuser', - }, - }; - - const output = await getDocFilesFromRepository( - new MockUrlReader(), - mockEntity, - ); - - expect(output).toBe('/tmp/testfolder'); - }); -}); diff --git a/plugins/techdocs-backend/src/index.ts b/plugins/techdocs-backend/src/index.ts index 8c65708017..5c57882788 100644 --- a/plugins/techdocs-backend/src/index.ts +++ b/plugins/techdocs-backend/src/index.ts @@ -15,4 +15,4 @@ */ export * from './service/router'; -export * from './techdocs'; +export * from '@backstage/techdocs-common'; diff --git a/plugins/techdocs-backend/src/service/helpers.test.ts b/plugins/techdocs-backend/src/service/helpers.test.ts new file mode 100644 index 0000000000..507373107d --- /dev/null +++ b/plugins/techdocs-backend/src/service/helpers.test.ts @@ -0,0 +1,26 @@ +/* + * Copyright 2020 Spotify AB + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import { getEntityNameFromUrlPath } from './helpers'; + +describe('getEntityNameFromUrlPath', () => { + it('should parse correctly', () => { + const path = 'default/Component/documented-component'; + const parsedEntity = getEntityNameFromUrlPath(path); + expect(parsedEntity).toHaveProperty('namespace', 'default'); + expect(parsedEntity).toHaveProperty('kind', 'Component'); + expect(parsedEntity).toHaveProperty('name', 'documented-component'); + }); +}); diff --git a/plugins/techdocs-backend/src/service/helpers.ts b/plugins/techdocs-backend/src/service/helpers.ts index 012061dc0a..fc80ab717d 100644 --- a/plugins/techdocs-backend/src/service/helpers.ts +++ b/plugins/techdocs-backend/src/service/helpers.ts @@ -13,126 +13,18 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -import Docker from 'dockerode'; -import { Logger } from 'winston'; -import { Entity } from '@backstage/catalog-model'; -import { - PreparerBuilder, - PublisherBase, - GeneratorBuilder, - PreparerBase, - GeneratorBase, -} from '../techdocs'; -import { BuildMetadataStorage } from '../storage'; -import { getLocationForEntity, getLastCommitTimestamp } from '../helpers'; +import { EntityName } from '@backstage/catalog-model'; +/** + * Using the path of the TechDocs page URL, return a structured EntityName type object with namespace, + * kind and name of the Entity. + * @param {string} path Example: default/Component/documented-component + */ +export const getEntityNameFromUrlPath = (path: string): EntityName => { + const [namespace, kind, name] = path.split('/'); -const getEntityId = (entity: Entity) => { - return `${entity.kind}:${entity.metadata.namespace ?? ''}:${ - entity.metadata.name - }`; + return { + namespace, + kind, + name, + }; }; - -type DocsBuilderArguments = { - preparers: PreparerBuilder; - generators: GeneratorBuilder; - publisher: PublisherBase; - entity: Entity; - logger: Logger; - dockerClient: Docker; -}; - -export class DocsBuilder { - private preparer: PreparerBase; - private generator: GeneratorBase; - private publisher: PublisherBase; - private entity: Entity; - private logger: Logger; - private dockerClient: Docker; - - constructor({ - preparers, - generators, - publisher, - entity, - logger, - dockerClient, - }: DocsBuilderArguments) { - this.preparer = preparers.get(entity); - this.generator = generators.get(entity); - this.publisher = publisher; - this.entity = entity; - this.logger = logger; - this.dockerClient = dockerClient; - } - - public async build() { - this.logger.info(`Running preparer on entity ${getEntityId(this.entity)}`); - const preparedDir = await this.preparer.prepare(this.entity); - - const parsedLocationAnnotation = getLocationForEntity(this.entity); - - this.logger.info(`Running generator on entity ${getEntityId(this.entity)}`); - const { resultDir } = await this.generator.run({ - directory: preparedDir, - dockerClient: this.dockerClient, - parsedLocationAnnotation, - }); - - this.logger.info(`Running publisher on entity ${getEntityId(this.entity)}`); - await this.publisher.publish({ - entity: this.entity, - directory: resultDir, - }); - - if (!this.entity.metadata.uid) { - throw new Error( - 'Trying to build documentation for entity not in service catalog', - ); - } - - new BuildMetadataStorage(this.entity.metadata.uid).storeBuildTimestamp(); - } - - public async docsUpToDate() { - if (!this.entity.metadata.uid) { - throw new Error( - 'Trying to build documentation for entity not in service catalog', - ); - } - - const buildMetadataStorage = new BuildMetadataStorage( - this.entity.metadata.uid, - ); - const { type, target } = getLocationForEntity(this.entity); - - // Unless docs are stored locally - const nonAgeCheckTypes = ['dir', 'file', 'url']; - if (!nonAgeCheckTypes.includes(type)) { - const lastCommit = await getLastCommitTimestamp(target, this.logger); - const storageTimeStamp = buildMetadataStorage.getTimestamp(); - - // Check if documentation source is newer than what we have - if (storageTimeStamp && storageTimeStamp >= lastCommit) { - this.logger.debug( - `Docs for entity ${getEntityId(this.entity)} is up to date.`, - ); - return true; - } - } - - // TODO: Better caching for URL. - if (type === 'url') { - const builtAt = buildMetadataStorage.getTimestamp(); - const now = Date.now(); - - if (builtAt > now - 1800000) { - return true; - } - } - - this.logger.debug( - `Docs for entity ${getEntityId(this.entity)} was outdated.`, - ); - return false; - } -} diff --git a/plugins/techdocs-backend/src/service/router.ts b/plugins/techdocs-backend/src/service/router.ts index 625252dd7e..a64f581a1e 100644 --- a/plugins/techdocs-backend/src/service/router.ts +++ b/plugins/techdocs-backend/src/service/router.ts @@ -24,15 +24,12 @@ import { GeneratorBuilder, PreparerBuilder, PublisherBase, - LocalPublish, -} from '../techdocs'; -import { - PluginEndpointDiscovery, - resolvePackagePath, -} from '@backstage/backend-common'; + getLocationForEntity, +} from '@backstage/techdocs-common'; +import { PluginEndpointDiscovery } from '@backstage/backend-common'; import { Entity } from '@backstage/catalog-model'; -import { DocsBuilder } from './helpers'; -import { getLocationForEntity } from '../helpers'; +import { getEntityNameFromUrlPath } from './helpers'; +import { DocsBuilder } from '../DocsBuilder'; type RouterOptions = { preparers: PreparerBuilder; @@ -45,11 +42,6 @@ type RouterOptions = { dockerClient: Docker; }; -const staticDocsDir = resolvePackagePath( - '@backstage/plugin-techdocs-backend', - 'static/docs', -); - export async function createRouter({ preparers, generators, @@ -62,24 +54,18 @@ export async function createRouter({ const router = Router(); router.get('/metadata/techdocs/*', async (req, res) => { - let storageUrl = config.getString('techdocs.storageUrl'); - if (publisher instanceof LocalPublish) { - storageUrl = new URL( - new URL(storageUrl).pathname, - await discovery.getBaseUrl('techdocs'), - ).toString(); - } + // path is `:namespace/:kind:/:name` const { '0': path } = req.params; + const entityName = getEntityNameFromUrlPath(path); - const metadataURL = `${storageUrl}/${path}/techdocs_metadata.json`; - - try { - const techdocsMetadata = await (await fetch(metadataURL)).json(); - res.send(techdocsMetadata); - } catch (err) { - logger.info(`Unable to get metadata for ${path} with error ${err}`); - throw new Error(`Unable to get metadata for ${path} with error ${err}`); - } + publisher + .fetchTechDocsMetadata(entityName) + .then(techdocsMetadataJson => { + res.send(techdocsMetadataJson); + }) + .catch(reason => { + res.status(500).send(`Unable to get Metadata. Reason: ${reason}`); + }); }); router.get('/metadata/entity/:namespace/:kind/:name', async (req, res) => { @@ -107,9 +93,8 @@ export async function createRouter({ }); router.get('/docs/:namespace/:kind/:name/*', async (req, res) => { - const storageUrl = config.getString('techdocs.storageUrl'); - const { kind, namespace, name } = req.params; + const storageUrl = config.getString('techdocs.storageUrl'); const catalogUrl = await discovery.getBaseUrl('catalog'); const triple = [kind, namespace, name].map(encodeURIComponent).join('/'); @@ -124,25 +109,80 @@ export async function createRouter({ const entity: Entity = await catalogRes.json(); - const docsBuilder = new DocsBuilder({ - preparers, - generators, - publisher, - dockerClient, - logger, - entity, - }); + let publisherType = ''; + try { + publisherType = config.getString('techdocs.publisher.type'); + } catch (err) { + throw new Error( + 'Unable to get techdocs.publisher.type in your app config. Set it to either ' + + "'local', 'googleGcs' or other support storage providers. Read more here " + + 'https://backstage.io/docs/features/techdocs/architecture', + ); + } - if (!(await docsBuilder.docsUpToDate())) { - await docsBuilder.build(); + // techdocs-backend will only try to build documentation for an entity if techdocs.builder is set to 'local' + // If set to 'external', it will only try to fetch and assume that an external process (e.g. CI/CD pipeline + // of the repository) is responsible for building and publishing documentation to the storage provider. + if (config.getString('techdocs.builder') === 'local') { + const docsBuilder = new DocsBuilder({ + preparers, + generators, + publisher, + dockerClient, + logger, + entity, + }); + if (publisherType === 'local') { + if (!(await docsBuilder.docsUpToDate())) { + await docsBuilder.build(); + } + } else if (publisherType === 'googleGcs') { + // This block should be valid for all external storage implementations. So no need to duplicate in future, + // add the publisher type in the list here. + if (!(await publisher.hasDocsBeenGenerated(entity))) { + logger.info( + 'No pre-generated documentation files found for the entity in the storage. Building docs...', + ); + await docsBuilder.build(); + // With a maximum of ~5 seconds wait, check if the files got published and if docs will be fetched + // on the user's page. If not, respond with a message asking them to check back later. + // The delay here is to make sure GCS registers newly uploaded files which is usually <1 second + let foundDocs = false; + for (let attempt = 0; attempt < 5; attempt++) { + if (await publisher.hasDocsBeenGenerated(entity)) { + foundDocs = true; + break; + } + await new Promise(r => setTimeout(r, 1000)); + } + if (!foundDocs) { + logger.error( + 'Published files are taking longer to show up in storage. Something went wrong.', + ); + res + .status(408) + .send( + 'Sorry! It is taking longer for the generated docs to show up in storage. Check back later.', + ); + return; + } + } else { + logger.info( + 'Found pre-generated docs for this entity. Serving them.', + ); + // TODO: re-trigger build for cache invalidation. + // Compare the date modified of the requested file on storage and compare it against + // the last modified or last commit timestamp in the repository. + // Without this, docs will not be re-built once they have been generated. + } + } } res.redirect(`${storageUrl}${req.path.replace('/docs', '')}`); }); - if (publisher instanceof LocalPublish) { - router.use('/static/docs', express.static(staticDocsDir)); - } + // Route middleware which serves files from the storage set in the publisher. + router.use('/static/docs', publisher.docsRouter()); return router; } diff --git a/plugins/techdocs-backend/src/service/standaloneServer.ts b/plugins/techdocs-backend/src/service/standaloneServer.ts index 4082a36172..c1c4dba8d1 100644 --- a/plugins/techdocs-backend/src/service/standaloneServer.ts +++ b/plugins/techdocs-backend/src/service/standaloneServer.ts @@ -27,8 +27,8 @@ import { DirectoryPreparer, Generators, TechdocsGenerator, - LocalPublish, -} from '../techdocs'; + Publisher, +} from '@backstage/techdocs-common'; import { ConfigReader } from '@backstage/config'; export interface ServerOptions { @@ -41,7 +41,18 @@ export async function startStandaloneServer( options: ServerOptions, ): Promise { const logger = options.logger.child({ service: 'techdocs-backend' }); - const config = ConfigReader.fromConfigs([]); + const config = ConfigReader.fromConfigs([ + { + context: '', + data: { + techdocs: { + publisher: { + type: 'local', + }, + }, + }, + }, + ]); const discovery = SingleHostDiscovery.fromConfig(config); logger.debug('Creating application...'); @@ -53,7 +64,7 @@ export async function startStandaloneServer( const techdocsGenerator = new TechdocsGenerator(logger, config); generators.register('techdocs', techdocsGenerator); - const publisher = new LocalPublish(logger, discovery); + const publisher = Publisher.fromConfig(config, logger, discovery); const dockerClient = new Docker(); diff --git a/plugins/techdocs-backend/src/techdocs/stages/publish/local.ts b/plugins/techdocs-backend/src/techdocs/stages/publish/local.ts deleted file mode 100644 index 464c4b90a0..0000000000 --- a/plugins/techdocs-backend/src/techdocs/stages/publish/local.ts +++ /dev/null @@ -1,82 +0,0 @@ -/* - * Copyright 2020 Spotify AB - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -import fs from 'fs-extra'; -import { Logger } from 'winston'; -import { Entity } from '@backstage/catalog-model'; -import { PublisherBase } from './types'; -import { - resolvePackagePath, - PluginEndpointDiscovery, -} from '@backstage/backend-common'; - -export class LocalPublish implements PublisherBase { - private readonly logger: Logger; - private readonly discovery: PluginEndpointDiscovery; - - constructor(logger: Logger, discovery: PluginEndpointDiscovery) { - this.logger = logger; - this.discovery = discovery; - } - - publish({ - entity, - directory, - }: { - entity: Entity; - directory: string; - }): - | Promise<{ - remoteUrl: string; - }> - | { remoteUrl: string } { - const entityNamespace = entity.metadata.namespace ?? 'default'; - - const publishDir = resolvePackagePath( - '@backstage/plugin-techdocs-backend', - 'static/docs', - entityNamespace, - entity.kind, - entity.metadata.name, - ); - - if (!fs.existsSync(publishDir)) { - this.logger.info(`Could not find ${publishDir}, creating the directory.`); - fs.mkdirSync(publishDir, { recursive: true }); - } - - return new Promise((resolve, reject) => { - fs.copy(directory, publishDir, err => { - if (err) { - this.logger.debug( - `Failed to copy docs from ${directory} to ${publishDir}`, - ); - reject(err); - } - - this.discovery - .getBaseUrl('techdocs') - .then(techdocsApiUrl => { - resolve({ - remoteUrl: `${techdocsApiUrl}/static/docs/${entity.metadata.name}`, - }); - }) - .catch(reason => { - reject(reason); - }); - }); - }); - } -} diff --git a/plugins/techdocs/README.md b/plugins/techdocs/README.md index 2e9365f6c6..c5e8fa6887 100644 --- a/plugins/techdocs/README.md +++ b/plugins/techdocs/README.md @@ -6,9 +6,7 @@ Set up Backstage and TechDocs by follow our guide on [Getting Started](../../doc ## Configuration -### Custom Storage URL - -TechDocs will try to read your documentation from the URL you have specified in the `techdocs storageUrl` in `app-config.yml`. +http://backstage.io/docs/features/techdocs/configuration ### TechDocs Storage Api diff --git a/plugins/techdocs/package.json b/plugins/techdocs/package.json index 215cb09f47..15325ed7bc 100644 --- a/plugins/techdocs/package.json +++ b/plugins/techdocs/package.json @@ -37,6 +37,7 @@ "@backstage/plugin-catalog": "^0.2.6", "@backstage/test-utils": "^0.1.5", "@backstage/theme": "^0.2.2", + "@backstage/techdocs-common": "^0.1.1", "@material-ui/core": "^4.11.0", "@material-ui/icons": "^4.9.1", "@material-ui/lab": "4.0.0-alpha.45", @@ -77,11 +78,68 @@ "visibility": "frontend" }, "storageUrl": { - "type": "string" + "type": "string", + "visibility": "backend" + }, + "generators": { + "type": "object", + "properties": { + "techdocs": { + "type": "string", + "visibility": "backend" + } + } + }, + "builder": { + "type": "string", + "visibility": "frontend" + }, + "publisher": { + "oneOf": [ + { + "type": "object", + "properties": { + "type": { + "type": "string", + "const": "local", + "visibility": "backend" + } + } + }, + { + "type": "object", + "properties": { + "type": { + "type": "string", + "const": "googleGcs", + "visibility": "backend" + }, + "googleGcs": { + "type": "object", + "properties": { + "credentials": { + "type": "string", + "visibility": "secret" + }, + "projectId": { + "type": "string", + "visibility": "secret" + }, + "bucketName": { + "type": "string", + "visibility": "secret" + } + } + } + } + } + ] } }, "required": [ - "requestUrl" + "requestUrl", + "storageUrl", + "builder" ] } } diff --git a/plugins/techdocs/src/reader/components/TechDocsNotFound.tsx b/plugins/techdocs/src/reader/components/TechDocsNotFound.tsx index 744242e343..cdacc1cb7e 100644 --- a/plugins/techdocs/src/reader/components/TechDocsNotFound.tsx +++ b/plugins/techdocs/src/reader/components/TechDocsNotFound.tsx @@ -15,18 +15,31 @@ */ import React from 'react'; -import { ErrorPage } from '@backstage/core'; +import { ErrorPage, useApi, configApiRef } from '@backstage/core'; type Props = { errorMessage?: string; }; export const TechDocsNotFound = ({ errorMessage }: Props) => { + const techdocsBuilder = useApi(configApiRef).getOptionalString( + 'techdocs.builder', + ); + + let additionalInfo = ''; + if (techdocsBuilder !== 'local') { + additionalInfo = + "Note that techdocs.builder is not set to 'local' in your config, which means this Backstage app will not " + + "generate docs if they are not found. Make sure the project's docs are generated and published by some external " + + "process (e.g. CI/CD pipeline). Or change techdocs.builder to 'local' to generate docs from this Backstage " + + 'instance.'; + } + return ( ); }; diff --git a/plugins/techdocs/src/reader/components/TechDocsPageHeader.tsx b/plugins/techdocs/src/reader/components/TechDocsPageHeader.tsx index d7a5a3c48f..ac9d21d319 100644 --- a/plugins/techdocs/src/reader/components/TechDocsPageHeader.tsx +++ b/plugins/techdocs/src/reader/components/TechDocsPageHeader.tsx @@ -16,7 +16,6 @@ import React from 'react'; import { AsyncState } from 'react-use/lib/useAsync'; -import { CircularProgress } from '@material-ui/core'; import CodeIcon from '@material-ui/icons/Code'; import { EntityName } from '@backstage/catalog-model'; import { Header, HeaderLabel, Link } from '@backstage/core'; @@ -86,7 +85,7 @@ export const TechDocsPageHeader = ({ return (
} + title={siteName ? siteName : '.'} pageTitleOverride={siteName || name} subtitle={ siteDescription && siteDescription !== 'None' ? siteDescription : '' diff --git a/yarn.lock b/yarn.lock index 43bfa27ad3..d174fe3c3e 100644 --- a/yarn.lock +++ b/yarn.lock @@ -2179,6 +2179,65 @@ query-string "^6.13.3" xcase "^2.0.1" +"@google-cloud/common@^3.5.0": + version "3.5.0" + resolved "https://registry.npmjs.org/@google-cloud/common/-/common-3.5.0.tgz#0959e769e8075a06eb0823cc567eef00fd0c2d02" + integrity sha512-10d7ZAvKhq47L271AqvHEd8KzJqGU45TY+rwM2Z3JHuB070FeTi7oJJd7elfrnKaEvaktw3hH2wKnRWxk/3oWQ== + dependencies: + "@google-cloud/projectify" "^2.0.0" + "@google-cloud/promisify" "^2.0.0" + arrify "^2.0.1" + duplexify "^4.1.1" + ent "^2.2.0" + extend "^3.0.2" + google-auth-library "^6.1.1" + retry-request "^4.1.1" + teeny-request "^7.0.0" + +"@google-cloud/paginator@^3.0.0": + version "3.0.5" + resolved "https://registry.npmjs.org/@google-cloud/paginator/-/paginator-3.0.5.tgz#9d6b96c421a89bd560c1bc2c197c7611ef21db6c" + integrity sha512-N4Uk4BT1YuskfRhKXBs0n9Lg2YTROZc6IMpkO/8DIHODtm5s3xY8K5vVBo23v/2XulY3azwITQlYWgT4GdLsUw== + dependencies: + arrify "^2.0.0" + extend "^3.0.2" + +"@google-cloud/projectify@^2.0.0": + version "2.0.1" + resolved "https://registry.npmjs.org/@google-cloud/projectify/-/projectify-2.0.1.tgz#13350ee609346435c795bbfe133a08dfeab78d65" + integrity sha512-ZDG38U/Yy6Zr21LaR3BTiiLtpJl6RkPS/JwoRT453G+6Q1DhlV0waNf8Lfu+YVYGIIxgKnLayJRfYlFJfiI8iQ== + +"@google-cloud/promisify@^2.0.0": + version "2.0.3" + resolved "https://registry.npmjs.org/@google-cloud/promisify/-/promisify-2.0.3.tgz#f934b5cdc939e3c7039ff62b9caaf59a9d89e3a8" + integrity sha512-d4VSA86eL/AFTe5xtyZX+ePUjE8dIFu2T8zmdeNBSa5/kNgXPCx/o/wbFNHAGLJdGnk1vddRuMESD9HbOC8irw== + +"@google-cloud/storage@^5.6.0": + version "5.6.0" + resolved "https://registry.npmjs.org/@google-cloud/storage/-/storage-5.6.0.tgz#bc6925c7970c375212a4da21c123298fc9665dec" + integrity sha512-nLcym8IuCzy1O7tNTXNFuMHfX900sTM3kSTqbKe7oFSoKUiaIM+FHuuuDimMMlieY6StA1xYNPRFFHz57Nv8YQ== + dependencies: + "@google-cloud/common" "^3.5.0" + "@google-cloud/paginator" "^3.0.0" + "@google-cloud/promisify" "^2.0.0" + arrify "^2.0.0" + compressible "^2.0.12" + date-and-time "^0.14.0" + duplexify "^4.0.0" + extend "^3.0.2" + gaxios "^4.0.0" + gcs-resumable-upload "^3.1.0" + get-stream "^6.0.0" + hash-stream-validation "^0.2.2" + mime "^2.2.0" + mime-types "^2.0.8" + onetime "^5.1.0" + p-limit "^3.0.1" + pumpify "^2.0.0" + snakeize "^0.1.0" + stream-events "^1.0.1" + xdg-basedir "^4.0.0" + "@graphql-codegen/cli@^1.17.7": version "1.17.10" resolved "https://registry.npmjs.org/@graphql-codegen/cli/-/cli-1.17.10.tgz#efebf9b887fdb94dd26dbf3eb1950e832efcda0e" @@ -6900,6 +6959,13 @@ abbrev@1: resolved "https://registry.npmjs.org/abbrev/-/abbrev-1.1.1.tgz#f8f2c887ad10bf67f634f005b6987fed3179aac8" integrity sha512-nne9/IiQ/hzIhY6pdDnbBtz7DjPTKrY00P/zvPSm5pOFkl6xuGrGnXn/VtTNNfNtAfZ9/1RtehkszU9qcTii0Q== +abort-controller@^3.0.0: + version "3.0.0" + resolved "https://registry.npmjs.org/abort-controller/-/abort-controller-3.0.0.tgz#eaf54d53b62bae4138e809ca225c8439a6efb392" + integrity sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg== + dependencies: + event-target-shim "^5.0.0" + abstract-logging@^2.0.0: version "2.0.0" resolved "https://registry.npmjs.org/abstract-logging/-/abstract-logging-2.0.0.tgz#08a85814946c98ef06f4256ad470aba1886d4490" @@ -7587,6 +7653,11 @@ arrify@^1.0.1: resolved "https://registry.npmjs.org/arrify/-/arrify-1.0.1.tgz#898508da2226f380df904728456849c1501a4b0d" integrity sha1-iYUI2iIm84DfkEcoRWhJwVAaSw0= +arrify@^2.0.0, arrify@^2.0.1: + version "2.0.1" + resolved "https://registry.npmjs.org/arrify/-/arrify-2.0.1.tgz#c9655e9331e0abcd588d2a7cad7e9956f66701fa" + integrity sha512-3duEwti880xqi4eAMN8AyR4a0ByT90zoYdLlevfrvU43vb0YZwZVfxOgxWrLXXXpyugL0hNZc9G6BiB5B3nUug== + asap@^2.0.0, asap@~2.0.3, asap@~2.0.6: version "2.0.6" resolved "https://registry.npmjs.org/asap/-/asap-2.0.6.tgz#e50347611d7e690943208bbdafebcbc2fb866d46" @@ -8187,7 +8258,7 @@ base64-js@^1.0.2, base64-js@^1.2.0: resolved "https://registry.npmjs.org/base64-js/-/base64-js-1.3.1.tgz#58ece8cb75dd07e71ed08c736abc5fac4dbf8df1" integrity sha512-mLQ4i2QO1ytvGWFWmcngKO//JXAQueZvwEKtjgQFM4jIK0kU+ytMfplL8j+n5mspOfjHwoAg+9yhb7BwAHm36g== -base64-js@^1.3.1: +base64-js@^1.3.0, base64-js@^1.3.1: version "1.5.1" resolved "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz#1b1b440160a5bf7ad40b650f095963481903930a" integrity sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA== @@ -8273,6 +8344,11 @@ big.js@^5.2.2: resolved "https://registry.npmjs.org/big.js/-/big.js-5.2.2.tgz#65f0af382f578bcdc742bd9c281e9cb2d7768328" integrity sha512-vyL2OymJxmarO8gxMr0mhChsO9QGwhynfuu4+MHTAW6czfq9humCB7rKpUjDd9YUiDPU4mzpyupFSvOClAwbmQ== +bignumber.js@^9.0.0: + version "9.0.1" + resolved "https://registry.npmjs.org/bignumber.js/-/bignumber.js-9.0.1.tgz#8d7ba124c882bfd8e43260c67475518d0689e4e5" + integrity sha512-IdZR9mh6ahOBv/hYGiXyVuyCetmGJhtYkqLBpTStdhEGjegpPlUawydyaF3pbIOFynJTpllEs+NP+CS9jKFLjA== + binary-extensions@^1.0.0: version "1.13.1" resolved "https://registry.npmjs.org/binary-extensions/-/binary-extensions-1.13.1.tgz#598afe54755b2868a5330d2aff9d4ebb53209b65" @@ -9535,7 +9611,7 @@ compress-commons@^4.0.0: normalize-path "^3.0.0" readable-stream "^3.6.0" -compressible@~2.0.16: +compressible@^2.0.12, compressible@~2.0.16: version "2.0.18" resolved "https://registry.npmjs.org/compressible/-/compressible-2.0.18.tgz#af53cca6b070d4c3c0750fbd77286a6d7cc46fba" integrity sha512-AF3r7P5dWxL8MxyITRMlORQNaOA2IkAFaTr4k7BUumjPtRpGDTZpl0Pb1XCO6JeDCBdp126Cgs9sMxqSjgYyRg== @@ -9629,7 +9705,7 @@ config-chain@^1.1.11: ini "^1.3.4" proto-list "~1.2.1" -configstore@^5.0.1: +configstore@^5.0.0, configstore@^5.0.1: version "5.0.1" resolved "https://registry.npmjs.org/configstore/-/configstore-5.0.1.tgz#d365021b5df4b98cdd187d6a3b0e3f6a7cc5ed96" integrity sha512-aMKprgk5YhBNyH25hj8wGt2+D52Sw1DRRIzqBwLp2Ya9mFmY8KPvvtvmna8SxVR9JMZ4kzMD68N22vlaRpkeFA== @@ -10594,6 +10670,11 @@ dataloader@2.0.0: resolved "https://registry.npmjs.org/dataloader/-/dataloader-2.0.0.tgz#41eaf123db115987e21ca93c005cd7753c55fe6f" integrity sha512-YzhyDAwA4TaQIhM5go+vCLmU0UikghC/t9DTQYZR2M/UvZ1MdOhPezSDZcjj9uqQJOMqjLcpWtyW2iNINdlatQ== +date-and-time@^0.14.0: + version "0.14.1" + resolved "https://registry.npmjs.org/date-and-time/-/date-and-time-0.14.1.tgz#969634697b78956fb66b8be6fb0f39fbd631f2f6" + integrity sha512-M4RggEH5OF2ZuCOxgOU67R6Z9ohjKbxGvAQz48vj53wLmL0bAgumkBvycR32f30pK+Og9pIR+RFDyChbaE4oLA== + date-fns@^1.27.2: version "1.30.1" resolved "https://registry.npmjs.org/date-fns/-/date-fns-1.30.1.tgz#2e71bf0b119153dbb4cc4e88d9ea5acfb50dc05c" @@ -11275,6 +11356,16 @@ duplexify@^3.4.2, duplexify@^3.6.0: readable-stream "^2.0.0" stream-shift "^1.0.0" +duplexify@^4.0.0, duplexify@^4.1.1: + version "4.1.1" + resolved "https://registry.npmjs.org/duplexify/-/duplexify-4.1.1.tgz#7027dc374f157b122a8ae08c2d3ea4d2d953aa61" + integrity sha512-DY3xVEmVHTv1wSzKNbwoU6nVjzI369Y6sPoqfYr0/xlx3IdX2n94xIszTcjPO8W8ZIv0Wb0PXNcjuZyT4wiICA== + dependencies: + end-of-stream "^1.4.1" + inherits "^2.0.3" + readable-stream "^3.1.1" + stream-shift "^1.0.0" + ecc-jsbn@~0.1.1: version "0.1.2" resolved "https://registry.npmjs.org/ecc-jsbn/-/ecc-jsbn-0.1.2.tgz#3a83a904e54353287874c564b7549386849a98c9" @@ -11283,7 +11374,7 @@ ecc-jsbn@~0.1.1: jsbn "~0.1.0" safer-buffer "^2.1.0" -ecdsa-sig-formatter@1.0.11: +ecdsa-sig-formatter@1.0.11, ecdsa-sig-formatter@^1.0.11: version "1.0.11" resolved "https://registry.npmjs.org/ecdsa-sig-formatter/-/ecdsa-sig-formatter-1.0.11.tgz#ae0f0fa2d85045ef14a817daa3ce9acd0489e5bf" integrity sha512-nagl3RYrbNv6kQkeJIpt6NJZy8twLB/2vtz6yN9Z4vRKHN4/QZJIEbqohALSgwKdnksuY3k5Addp5lg8sVoVcQ== @@ -11432,6 +11523,11 @@ enquirer@^2.3.0, enquirer@^2.3.5: dependencies: ansi-colors "^4.1.1" +ent@^2.2.0: + version "2.2.0" + resolved "https://registry.npmjs.org/ent/-/ent-2.2.0.tgz#e964219325a21d05f44466a2f686ed6ce5f5dd1d" + integrity sha1-6WQhkyWiHQX0RGai9obtbOX13R0= + entities@^1.1.1, entities@^1.1.2: version "1.1.2" resolved "https://registry.npmjs.org/entities/-/entities-1.1.2.tgz#bdfa735299664dfafd34529ed4f8522a275fea56" @@ -11980,6 +12076,11 @@ event-stream@=3.3.4: stream-combiner "~0.0.4" through "~2.3.1" +event-target-shim@^5.0.0: + version "5.0.1" + resolved "https://registry.npmjs.org/event-target-shim/-/event-target-shim-5.0.1.tgz#5d4d3ebdf9583d63a5333ce2deb7480ab2b05789" + integrity sha512-i/2XbnSz/uxRCU6+NdVJgKWDTM427+MqYbkQzD321DuCQJUqOuJKIA0IM2+W2xtYHdKOmZ4dR6fExsd4SXL+WQ== + eventemitter2@^6.4.2: version "6.4.3" resolved "https://registry.npmjs.org/eventemitter2/-/eventemitter2-6.4.3.tgz#35c563619b13f3681e7eb05cbdaf50f56ba58820" @@ -12207,7 +12308,7 @@ extend-shallow@^3.0.0, extend-shallow@^3.0.2: assign-symbols "^1.0.0" is-extendable "^1.0.1" -extend@3.0.2, extend@^3.0.0, extend@~3.0.2: +extend@3.0.2, extend@^3.0.0, extend@^3.0.2, extend@~3.0.2: version "3.0.2" resolved "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz#f8b1136b4071fbd8eb140aff858b1019ec2915fa" integrity sha512-fjquC59cD7CyW6urNXK0FBufkZcoiGG80wTuPujX590cB5Ttln20E2UB4S/WARVqhXffZl2LNgS+gQdPIIim/g== @@ -12334,6 +12435,11 @@ fast-shallow-equal@^1.0.0: resolved "https://registry.npmjs.org/fast-shallow-equal/-/fast-shallow-equal-1.0.0.tgz#d4dcaf6472440dcefa6f88b98e3251e27f25628b" integrity sha512-HPtaa38cPgWvaCFmRNhlc6NG7pv6NUHqjPgVAkWGoB9mQMwYB27/K0CvOM5Czy+qpT3e8XJ6Q4aPAnzpNpzNaw== +fast-text-encoding@^1.0.0: + version "1.0.3" + resolved "https://registry.npmjs.org/fast-text-encoding/-/fast-text-encoding-1.0.3.tgz#ec02ac8e01ab8a319af182dae2681213cfe9ce53" + integrity sha512-dtm4QZH9nZtcDt8qJiOH9fcQd1NAgi+K1O2DbE6GG1PPCK/BWfOH3idCTRQ4ImXRUOyopDEgDEnVEE7Y/2Wrig== + fastest-stable-stringify@^1.0.1: version "1.0.1" resolved "https://registry.npmjs.org/fastest-stable-stringify/-/fastest-stable-stringify-1.0.1.tgz#9122d406d4c9d98bea644a6b6853d5874b87b028" @@ -12930,6 +13036,49 @@ gauge@~2.7.3: strip-ansi "^3.0.1" wide-align "^1.1.0" +gaxios@^3.0.0: + version "3.2.0" + resolved "https://registry.npmjs.org/gaxios/-/gaxios-3.2.0.tgz#11b6f0e8fb08d94a10d4d58b044ad3bec6dd486a" + integrity sha512-+6WPeVzPvOshftpxJwRi2Ozez80tn/hdtOUag7+gajDHRJvAblKxTFSSMPtr2hmnLy7p0mvYz0rMXLBl8pSO7Q== + dependencies: + abort-controller "^3.0.0" + extend "^3.0.2" + https-proxy-agent "^5.0.0" + is-stream "^2.0.0" + node-fetch "^2.3.0" + +gaxios@^4.0.0: + version "4.0.1" + resolved "https://registry.npmjs.org/gaxios/-/gaxios-4.0.1.tgz#bc7b205a89d883452822cc75e138620c35e3291e" + integrity sha512-jOin8xRZ/UytQeBpSXFqIzqU7Fi5TqgPNLlUsSB8kjJ76+FiGBfImF8KJu++c6J4jOldfJUtt0YmkRj2ZpSHTQ== + dependencies: + abort-controller "^3.0.0" + extend "^3.0.2" + https-proxy-agent "^5.0.0" + is-stream "^2.0.0" + node-fetch "^2.3.0" + +gcp-metadata@^4.2.0: + version "4.2.1" + resolved "https://registry.npmjs.org/gcp-metadata/-/gcp-metadata-4.2.1.tgz#31849fbcf9025ef34c2297c32a89a1e7e9f2cd62" + integrity sha512-tSk+REe5iq/N+K+SK1XjZJUrFPuDqGZVzCy2vocIHIGmPlTGsa8owXMJwGkrXr73NO0AzhPW4MF2DEHz7P2AVw== + dependencies: + gaxios "^4.0.0" + json-bigint "^1.0.0" + +gcs-resumable-upload@^3.1.0: + version "3.1.1" + resolved "https://registry.npmjs.org/gcs-resumable-upload/-/gcs-resumable-upload-3.1.1.tgz#67c766a0555d6a352f9651b7603337207167d0de" + integrity sha512-RS1osvAicj9+MjCc6jAcVL1Pt3tg7NK2C2gXM5nqD1Gs0klF2kj5nnAFSBy97JrtslMIQzpb7iSuxaG8rFWd2A== + dependencies: + abort-controller "^3.0.0" + configstore "^5.0.0" + extend "^3.0.2" + gaxios "^3.0.0" + google-auth-library "^6.0.0" + pumpify "^2.0.0" + stream-events "^1.0.4" + generic-names@^2.0.1: version "2.0.1" resolved "https://registry.npmjs.org/generic-names/-/generic-names-2.0.1.tgz#f8a378ead2ccaa7a34f0317b05554832ae41b872" @@ -13034,6 +13183,11 @@ get-stream@^5.0.0, get-stream@^5.1.0: dependencies: pump "^3.0.0" +get-stream@^6.0.0: + version "6.0.0" + resolved "https://registry.npmjs.org/get-stream/-/get-stream-6.0.0.tgz#3e0012cb6827319da2706e601a1583e8629a6718" + integrity sha512-A1B3Bh1UmL0bidM/YX2NsCOTnGJePL9rO/M+Mw3m9f2gUpfokS0hi5Eah0WSUEWZdZhIZtMjkIYS7mDfOqNHbg== + get-value@^2.0.3, get-value@^2.0.6: version "2.0.6" resolved "https://registry.npmjs.org/get-value/-/get-value-2.0.6.tgz#dc15ca1c672387ca76bd37ac0a395ba2042a2c28" @@ -13328,6 +13482,28 @@ good-listener@^1.2.2: dependencies: delegate "^3.1.2" +google-auth-library@^6.0.0, google-auth-library@^6.1.1: + version "6.1.3" + resolved "https://registry.npmjs.org/google-auth-library/-/google-auth-library-6.1.3.tgz#39d868140b70d0c4b32c6f6d8f4ccc1400d84dca" + integrity sha512-m9mwvY3GWbr7ZYEbl61isWmk+fvTmOt0YNUfPOUY2VH8K5pZlAIWJjxEi0PqR3OjMretyiQLI6GURMrPSwHQ2g== + dependencies: + arrify "^2.0.0" + base64-js "^1.3.0" + ecdsa-sig-formatter "^1.0.11" + fast-text-encoding "^1.0.0" + gaxios "^4.0.0" + gcp-metadata "^4.2.0" + gtoken "^5.0.4" + jws "^4.0.0" + lru-cache "^6.0.0" + +google-p12-pem@^3.0.3: + version "3.0.3" + resolved "https://registry.npmjs.org/google-p12-pem/-/google-p12-pem-3.0.3.tgz#673ac3a75d3903a87f05878f3c75e06fc151669e" + integrity sha512-wS0ek4ZtFx/ACKYF3JhyGe5kzH7pgiQ7J5otlumqR9psmWMYc+U9cErKlCYVYHoUaidXHdZ2xbo34kB+S+24hA== + dependencies: + node-forge "^0.10.0" + got@^10.7.0: version "10.7.0" resolved "https://registry.npmjs.org/got/-/got-10.7.0.tgz#62889dbcd6cca32cd6a154cc2d0c6895121d091f" @@ -13593,6 +13769,16 @@ growly@^1.3.0: resolved "https://registry.npmjs.org/growly/-/growly-1.3.0.tgz#f10748cbe76af964b7c96c93c6bcc28af120c081" integrity sha1-8QdIy+dq+WS3yWyTxrzCivEgwIE= +gtoken@^5.0.4: + version "5.1.0" + resolved "https://registry.npmjs.org/gtoken/-/gtoken-5.1.0.tgz#4ba8d2fc9a8459098f76e7e8fd7beaa39fda9fe4" + integrity sha512-4d8N6Lk8TEAHl9vVoRVMh9BNOKWVgl2DdNtr3428O75r3QFrF/a5MMu851VmK0AA8+iSvbwRv69k5XnMLURGhg== + dependencies: + gaxios "^4.0.0" + google-p12-pem "^3.0.3" + jws "^4.0.0" + mime "^2.2.0" + gud@^1.0.0: version "1.0.0" resolved "https://registry.npmjs.org/gud/-/gud-1.0.0.tgz#a489581b17e6a70beca9abe3ae57de7a499852c0" @@ -13724,6 +13910,11 @@ hash-base@^3.0.0: inherits "^2.0.1" safe-buffer "^5.0.1" +hash-stream-validation@^0.2.2: + version "0.2.4" + resolved "https://registry.npmjs.org/hash-stream-validation/-/hash-stream-validation-0.2.4.tgz#ee68b41bf822f7f44db1142ec28ba9ee7ccb7512" + integrity sha512-Gjzu0Xn7IagXVkSu9cSFuK1fqzwtLwFhNhVL8IFJijRNMgUttFbBSIAzKuSIrsFMO1+g1RlsoN49zPIbwPDMGQ== + hash.js@^1.0.0, hash.js@^1.0.3: version "1.1.7" resolved "https://registry.npmjs.org/hash.js/-/hash.js-1.1.7.tgz#0babca538e8d4ee4a0f8988d68866537a003cf42" @@ -14016,7 +14207,7 @@ http-proxy-agent@^2.1.0: agent-base "4" debug "3.1.0" -http-proxy-agent@^4.0.1: +http-proxy-agent@^4.0.0, http-proxy-agent@^4.0.1: version "4.0.1" resolved "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-4.0.1.tgz#8a8c8ef7f5932ccf953c296ca8291b95aa74aa3a" integrity sha512-k0zdNgqWTGA6aeIRVpvfVob4fL52dTfaehylg0Y4UvSySvOq/Y+BOyPrgpUrA7HylqvU8vIZGsRuXmspskV0Tg== @@ -15694,6 +15885,13 @@ jsesc@~0.5.0: resolved "https://registry.npmjs.org/jsesc/-/jsesc-0.5.0.tgz#e7dee66e35d6fc16f710fe91d5cf69f70f08911d" integrity sha1-597mbjXW/Bb3EP6R1c9p9w8IkR0= +json-bigint@^1.0.0: + version "1.0.0" + resolved "https://registry.npmjs.org/json-bigint/-/json-bigint-1.0.0.tgz#ae547823ac0cad8398667f8cd9ef4730f5b01ff1" + integrity sha512-SiPv/8VpZuWbvLSMtTDU8hEfrZWg/mH/nV/b4o0CYbSxu1UIQPLdwKOCIyLQX+VIPO5vrLX3i8qtqFyhdPSUSQ== + dependencies: + bignumber.js "^9.0.0" + json-buffer@3.0.0: version "3.0.0" resolved "https://registry.npmjs.org/json-buffer/-/json-buffer-3.0.0.tgz#5b1f397afc75d677bde8bcfc0e47e1f9a3d9a898" @@ -15994,6 +16192,15 @@ jwa@^1.4.1: ecdsa-sig-formatter "1.0.11" safe-buffer "^5.0.1" +jwa@^2.0.0: + version "2.0.0" + resolved "https://registry.npmjs.org/jwa/-/jwa-2.0.0.tgz#a7e9c3f29dae94027ebcaf49975c9345593410fc" + integrity sha512-jrZ2Qx916EA+fq9cEAeCROWPTfCwi1IVHqT2tapuqLEVVDKFDENFw1oL+MwrTvH6msKxsd1YTDVw6uKEcsrLEA== + dependencies: + buffer-equal-constant-time "1.0.1" + ecdsa-sig-formatter "1.0.11" + safe-buffer "^5.0.1" + jws@^3.2.2: version "3.2.2" resolved "https://registry.npmjs.org/jws/-/jws-3.2.2.tgz#001099f3639468c9414000e99995fa52fb478304" @@ -16002,6 +16209,14 @@ jws@^3.2.2: jwa "^1.4.1" safe-buffer "^5.0.1" +jws@^4.0.0: + version "4.0.0" + resolved "https://registry.npmjs.org/jws/-/jws-4.0.0.tgz#2d4e8cf6a318ffaa12615e9dec7e86e6c97310f4" + integrity sha512-KDncfTmOZoOMTFG4mBlG0qUIOlc03fmzH+ru6RgYVZhPkyiy/92Owlt/8UEN+a4TXR1FQetfIpJE8ApdvdVxTg== + dependencies: + jwa "^2.0.0" + safe-buffer "^5.0.1" + jwt-decode@*, jwt-decode@^3.1.0: version "3.1.2" resolved "https://registry.npmjs.org/jwt-decode/-/jwt-decode-3.1.2.tgz#3fb319f3675a2df0c2895c8f5e9fa4b67b04ed59" @@ -17262,7 +17477,7 @@ mime-db@1.44.0, "mime-db@>= 1.43.0 < 2": resolved "https://registry.npmjs.org/mime-db/-/mime-db-1.44.0.tgz#fa11c5eb0aca1334b4233cb4d52f10c5a6272f92" integrity sha512-/NOTfLrsPBVeH7YtFPgsVWveuL+4SjjYxaQ1xtM1KMFj7HdxlBlxeyNLzhyJVx7r4rZGJAZ/6lkKCitSc/Nmpg== -mime-types@^2.1.12, mime-types@^2.1.26, mime-types@~2.1.17, mime-types@~2.1.19, mime-types@~2.1.24: +mime-types@^2.0.8, mime-types@^2.1.12, mime-types@^2.1.26, mime-types@~2.1.17, mime-types@~2.1.19, mime-types@~2.1.24: version "2.1.27" resolved "https://registry.npmjs.org/mime-types/-/mime-types-2.1.27.tgz#47949f98e279ea53119f5722e0f34e529bec009f" integrity sha512-JIhqnCasI9yD+SsmkquHBxTSEuZdQX5BuQnS2Vc7puQQQ+8yiP5AY5uWhpdv4YL4VM5c6iliiYWPgJ/nJQLp7w== @@ -17274,6 +17489,11 @@ mime@1.6.0, mime@^1.4.1: resolved "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz#32cd9e5c64553bd58d19a568af452acff04981b1" integrity sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg== +mime@^2.2.0: + version "2.4.6" + resolved "https://registry.npmjs.org/mime/-/mime-2.4.6.tgz#e5b407c90db442f2beb5b162373d07b69affa4d1" + integrity sha512-RZKhC3EmpBchfTGBVb8fb+RL2cWyw/32lshnsETttkBAyAUXSGHxbEJWWRXc751DrIxG1q04b8QwMbAwkRPpUA== + mime@^2.3.1, mime@^2.4.4: version "2.4.4" resolved "https://registry.npmjs.org/mime/-/mime-2.4.4.tgz#bd7b91135fc6b01cde3e9bae33d659b63d8857e5" @@ -18582,6 +18802,13 @@ p-limit@^2.0.0, p-limit@^2.2.0: dependencies: p-try "^2.0.0" +p-limit@^3.0.1: + version "3.1.0" + resolved "https://registry.npmjs.org/p-limit/-/p-limit-3.1.0.tgz#e1daccbe78d0d1388ca18c64fea38e3e57e3706b" + integrity sha512-TYOanM3wGwNGsZN2cVTYPArw454xnXj5qmWF1bEoAc4+cU/ol7GVh7odevjp1FNHduHc3KZMcFduxU5Xc6uJRQ== + dependencies: + yocto-queue "^0.1.0" + p-locate@^2.0.0: version "2.0.0" resolved "https://registry.npmjs.org/p-locate/-/p-locate-2.0.0.tgz#20a0103b222a70c8fd39cc2e580680f3dde5ec43" @@ -20110,6 +20337,15 @@ pumpify@^1.3.3: inherits "^2.0.3" pump "^2.0.0" +pumpify@^2.0.0: + version "2.0.1" + resolved "https://registry.npmjs.org/pumpify/-/pumpify-2.0.1.tgz#abfc7b5a621307c728b551decbbefb51f0e4aa1e" + integrity sha512-m7KOje7jZxrmutanlkS1daj1dS6z6BgslzOXmcSEpIlCxM3VJH7lG5QLeck/6hgF6F4crFf01UtQmNsJfweTAw== + dependencies: + duplexify "^4.1.1" + inherits "^2.0.3" + pump "^3.0.0" + punycode@1.3.2: version "1.3.2" resolved "https://registry.npmjs.org/punycode/-/punycode-1.3.2.tgz#9653a036fb7c1ee42342f2325cceefea3926c48d" @@ -21598,6 +21834,13 @@ ret@~0.1.10: resolved "https://registry.npmjs.org/ret/-/ret-0.1.15.tgz#b8a4825d5bdb1fc3f6f53c2bc33f81388681c7bc" integrity sha512-TTlYpa+OL+vMMNG24xSlQGEJ3B/RzEfUlLct7b5G/ytav+wPrplCpVMFuwzXbkecJrb6IYo1iFb0S9v37754mg== +retry-request@^4.1.1: + version "4.1.3" + resolved "https://registry.npmjs.org/retry-request/-/retry-request-4.1.3.tgz#d5f74daf261372cff58d08b0a1979b4d7cab0fde" + integrity sha512-QnRZUpuPNgX0+D1xVxul6DbJ9slvo4Rm6iV/dn63e048MvGbUZiKySVt6Tenp04JqmchxjiLltGerOJys7kJYQ== + dependencies: + debug "^4.1.1" + retry@0.12.0, retry@^0.12.0: version "0.12.0" resolved "https://registry.npmjs.org/retry/-/retry-0.12.0.tgz#1b42a6266a21f07421d1b0b54b7dc167b01c013b" @@ -22262,6 +22505,11 @@ smartwrap@^1.2.3: wcwidth "^1.0.1" yargs "^15.1.0" +snakeize@^0.1.0: + version "0.1.0" + resolved "https://registry.npmjs.org/snakeize/-/snakeize-0.1.0.tgz#10c088d8b58eb076b3229bb5a04e232ce126422d" + integrity sha1-EMCI2LWOsHazIpu1oE4jLOEmQi0= + snapdragon-node@^2.0.1: version "2.1.1" resolved "https://registry.npmjs.org/snapdragon-node/-/snapdragon-node-2.1.1.tgz#6c175f86ff14bdb0724563e8f3c1b021a286853b" @@ -22713,6 +22961,13 @@ stream-each@^1.1.0: end-of-stream "^1.1.0" stream-shift "^1.0.0" +stream-events@^1.0.1, stream-events@^1.0.4, stream-events@^1.0.5: + version "1.0.5" + resolved "https://registry.npmjs.org/stream-events/-/stream-events-1.0.5.tgz#bbc898ec4df33a4902d892333d47da9bf1c406d5" + integrity sha512-E1GUzBSgvct8Jsb3v2X15pjzN1tYebtbLaMg+eBOUOAxgbLoSbT2NS91ckc5lJD1KfLjId+jXJRgo0qnV5Nerg== + dependencies: + stubs "^3.0.0" + stream-http@^2.7.2: version "2.8.3" resolved "https://registry.npmjs.org/stream-http/-/stream-http-2.8.3.tgz#b2d242469288a5a27ec4fe8933acf623de6514fc" @@ -22992,6 +23247,11 @@ strong-log-transformer@^2.0.0: minimist "^1.2.0" through "^2.3.4" +stubs@^3.0.0: + version "3.0.0" + resolved "https://registry.npmjs.org/stubs/-/stubs-3.0.0.tgz#e8d2ba1fa9c90570303c030b6900f7d5f89abe5b" + integrity sha1-6NK6H6nJBXAwPAMLaQD31fiavls= + style-inject@^0.3.0: version "0.3.0" resolved "https://registry.npmjs.org/style-inject/-/style-inject-0.3.0.tgz#d21c477affec91811cc82355832a700d22bf8dd3" @@ -23361,6 +23621,17 @@ tarn@^3.0.1: resolved "https://registry.npmjs.org/tarn/-/tarn-3.0.1.tgz#ebac2c6dbc6977d34d4526e0a7814200386a8aec" integrity sha512-6usSlV9KyHsspvwu2duKH+FMUhqJnAh6J5J/4MITl8s94iSUQTLkJggdiewKv4RyARQccnigV48Z+khiuVZDJw== +teeny-request@^7.0.0: + version "7.0.1" + resolved "https://registry.npmjs.org/teeny-request/-/teeny-request-7.0.1.tgz#bdd41fdffea5f8fbc0d29392cb47bec4f66b2b4c" + integrity sha512-sasJmQ37klOlplL4Ia/786M5YlOcoLGQyq2TE4WHSRupbAuDaQW0PfVxV4MtdBtRJ4ngzS+1qim8zP6Zp35qCw== + dependencies: + http-proxy-agent "^4.0.0" + https-proxy-agent "^5.0.0" + node-fetch "^2.6.1" + stream-events "^1.0.5" + uuid "^8.0.0" + telejson@^5.0.2: version "5.0.2" resolved "https://registry.npmjs.org/telejson/-/telejson-5.0.2.tgz#ed1e64be250cc1c757a53c19e1740b49832b3d51" @@ -25448,6 +25719,11 @@ yn@^4.0.0: resolved "https://registry.npmjs.org/yn/-/yn-4.0.0.tgz#611480051ea43b510da1dfdbe177ed159f00a979" integrity sha512-huWiiCS4TxKc4SfgmTwW1K7JmXPPAmuXWYy4j9qjQo4+27Kni8mGhAAi1cloRWmBe2EqcLgt3IGqQoRL/MtPgg== +yocto-queue@^0.1.0: + version "0.1.0" + resolved "https://registry.npmjs.org/yocto-queue/-/yocto-queue-0.1.0.tgz#0294eb3dee05028d31ee1a5fa2c556a6aaf10a1b" + integrity sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q== + yup@^0.29.3: version "0.29.3" resolved "https://registry.npmjs.org/yup/-/yup-0.29.3.tgz#69a30fd3f1c19f5d9e31b1cf1c2b851ce8045fea"