Merge pull request #9567 from Niksko/docs-build-strategy

Add a new interface: DocsBuildStrategy

.changeset/dull-months-knock.md

@@ -0,0 +1,5 @@
+---
+'@backstage/plugin-techdocs-backend': patch
+---
+
+Added a new interface that allows for customization of when to build techdocs

docs/features/techdocs/concepts.md

@@ -46,6 +46,24 @@ between `techdocs-backend` and the storage)
 
 [TechDocs Backend](https://github.com/backstage/backstage/tree/master/plugins/techdocs-backend)
 
+## TechDocs Build Strategy
+
+To accommodate more complex logic surrounding whether or not to build TechDocs, the TechDocs backend
+supports selecting a Build Strategy.
+The Build Strategy is responsible for deciding whether the documentation requested should be built locally
+by the TechDocs backend or not.
+Customization of the Build Strategy allows for more complex behaviour regarding whether the TechDocs backend
+is responsible for building TechDocs, whether an external process is responsible, or whether a combination
+of local builds and an external process is responsible, on an entity-by-entity basis.
+
+The default Build Strategy results in the TechDocs backend building documentation locally if the
+`techdocs.builder` configuration option is set to `'local'`, and skipping any building otherwise.
+However any logic that satisfies the Build Strategy interface can be implemented, using the Backstage
+config as well as the entity being processed to make a decision.
+
+For an example of how the Build Strategy can be used to implement a 'hybrid' build model, refer to
+the [How to implement a hybrid build strategy](./how-to-guides.md#how-to-implement-a-hybrid-build-strategy) guide.
+
 ## TechDocs Container
 
 The TechDocs container is a Docker container available at

docs/features/techdocs/configuration.md

@@ -38,11 +38,16 @@ techdocs:
     pullImage: true
 
   # techdocs.builder can be either 'local' or 'external.
-  # If builder is set to 'local' and you open a TechDocs page, techdocs-backend will try to generate the docs, publish to storage
-  # and show the generated docs afterwords. This is the "Basic" setup of the TechDocs Architecture.
-  # If builder is set to 'external', techdocs-backend will only fetch the docs and will NOT try to generate and publish. In this case of 'external',
-  # we assume that docs are being built by an external process (e.g. in the CI/CD pipeline of the repository). This is the "Recommended" setup of
-  # the architecture. Read more here https://backstage.io/docs/features/techdocs/architecture
+  # Using the default build strategy, if builder is set to 'local' and you open a TechDocs page,
+  # techdocs-backend will try to generate the docs, publish to storage and show the generated docs afterwords.
+  # This is the "Basic" setup of the TechDocs Architecture.
+  # Using the default build strategy, if builder is set to 'external' (or anything other than 'local'), techdocs-backend
+  # will only fetch the docs and will NOT try to generate and publish.
+  # In this case, we assume that docs are being built by an external process (e.g. in the CI/CD pipeline of the repository).
+  # This is the "Recommended" setup of the architecture.
+  # Note that custom build strategies may alter this behaviour.
+  # Read more about the "Basic" and "Recommended" setups here https://backstage.io/docs/features/techdocs/architecture
+  # Read more about build strategies here: https://backstage.io/docs/features/techdocs/concepts#techdocs-build-strategy
 
   builder: 'local'
 

docs/features/techdocs/how-to-guides.md

@@ -538,3 +538,58 @@ Done! Now you have a support of the following diagrams along with mermaid:
 - `Vega`
 - `Vega-Lite`
 - `WaveDrom`
+
+## How to implement a hybrid build strategy
+
+One limitation of the [Recommended deployment](./architecture.md#recommended-deployment) is that
+the experience for users requires modifying their CI/CD process to publish
+their TechDocs. For some users, this may be unnecessary, and provides a barrier
+to entry for onboarding users to Backstage. However, a purely local TechDocs
+build restricts TechDocs creators to using the tooling provided in Backstage,
+as well as the plugins and features provided in the Backstage-included `mkdocs`
+installation.
+
+To accommodate both of these use-cases, users can implement a custom [Build Strategy](./concepts.md#techdocs-build-strategy)
+with logic to encode which TechDocs should be built locally, and which will be
+built externally.
+
+To achieve this hybrid build model:
+
+1. In your Backstage instance's `app-config.yaml`, set `techdocs.builder` to
+   `'local'`. This ensures that Backstage will build docs for users who want the
+   'out-of-the-box' experience.
+2. Configure external storage of TechDocs as normal for a production deployment.
+   This allows Backstage to publish documentation to your storage, as well as
+   allowing other users to publish documentation from their CI/CD pipelines.
+3. Create a custom build strategy, that implements the `DocsBuildStrategy` interface,
+   and which implements your custom logic for determining whether to build docs for
+   a given entity.
+   For example, to only build docs when an entity has the `company.com/techdocs-builder`
+   annotation set to `'local'`:
+
+   ```typescript
+   export class AnnotationBasedBuildStrategy {
+     private readonly config: Config;
+
+     constructor(config: Config) {
+       this.config = config;
+     }
+
+     async shouldBuild(_: Entity): Promise<boolean> {
+       return (
+         this.entity.metadata?.annotations?.['company.com/techdocs-builder'] ===
+         'local'
+       );
+     }
+   }
+   ```
+
+4. Pass an instance of this Build Strategy as the `docsBuildStrategy` parameter of the
+   TechDocs backend `createRouter` method.
+
+Users should now be able to choose to have their documentation built and published by
+the TechDocs backend by adding the `company.com/techdocs-builder` annotation to their
+entity. If the value of this annotation is `'local'`, the TechDocs backend will build
+and publish the documentation for them. If the value of the `company.com/techdocs-builder`
+annotation is anything other than `'local'`, the user is responsible for publishing
+documentation to the appropriate location in the TechDocs external storage.

plugins/techdocs-backend/api-report.md

@@ -6,6 +6,7 @@
 import { CatalogApi } from '@backstage/catalog-client';
 import { Config } from '@backstage/config';
 import { DocumentCollator } from '@backstage/search-common';
+import { Entity } from '@backstage/catalog-model';
 import express from 'express';
 import { GeneratorBuilder } from '@backstage/techdocs-common';
 import { Knex } from 'knex';
@@ -41,6 +42,12 @@ export class DefaultTechDocsCollator implements DocumentCollator {
   readonly visibilityPermission: Permission;
 }
 
+// @public
+export interface DocsBuildStrategy {
+  // (undocumented)
+  shouldBuild(params: ShouldBuildParameters): Promise<boolean>;
+}
+
 // @public
 export type OutOfTheBoxDeploymentOptions = {
   preparers: PreparerBuilder;
@@ -51,6 +58,7 @@ export type OutOfTheBoxDeploymentOptions = {
   database?: Knex;
   config: Config;
   cache: PluginCacheManager;
+  docsBuildStrategy?: DocsBuildStrategy;
 };
 
 // @public
@@ -60,6 +68,7 @@ export type RecommendedDeploymentOptions = {
   discovery: PluginEndpointDiscovery;
   config: Config;
   cache: PluginCacheManager;
+  docsBuildStrategy?: DocsBuildStrategy;
 };
 
 // @public
@@ -67,6 +76,11 @@ export type RouterOptions =
   | RecommendedDeploymentOptions
   | OutOfTheBoxDeploymentOptions;
 
+// @public
+export type ShouldBuildParameters = {
+  entity: Entity;
+};
+
 // @public
 export type TechDocsCollatorOptions = {
   discovery: PluginEndpointDiscovery;

plugins/techdocs-backend/src/index.ts

@@ -25,6 +25,8 @@ export type {
   RouterOptions,
   RecommendedDeploymentOptions,
   OutOfTheBoxDeploymentOptions,
+  DocsBuildStrategy,
+  ShouldBuildParameters,
 } from './service';
 
 export { DefaultTechDocsCollator } from './search';

plugins/techdocs-backend/src/service/DocsBuildStrategy.test.ts

@@ -0,0 +1,65 @@
+/*
+ * Copyright 2022 The Backstage Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { DefaultDocsBuildStrategy } from './DocsBuildStrategy';
+import { ConfigReader } from '@backstage/config';
+
+const MockedConfigReader = ConfigReader as jest.MockedClass<
+  typeof ConfigReader
+>;
+
+jest.mock('@backstage/config');
+
+describe('DefaultDocsBuildStrategy', () => {
+  const entity = {
+    apiVersion: 'backstage.io/v1alpha1',
+    kind: 'Component',
+    metadata: {
+      uid: '0',
+      name: 'test',
+    },
+  };
+
+  const config = new ConfigReader({});
+
+  beforeEach(() => {
+    jest.resetAllMocks();
+  });
+
+  describe('shouldBuild', () => {
+    it('should return true when techdocs.build is set to local', async () => {
+      const defaultDocsBuildStrategy =
+        DefaultDocsBuildStrategy.fromConfig(config);
+
+      MockedConfigReader.prototype.getString.mockReturnValue('local');
+
+      const result = await defaultDocsBuildStrategy.shouldBuild({ entity });
+
+      expect(result).toBe(true);
+    });
+
+    it('should return false when techdocs.build is set to external', async () => {
+      const defaultDocsBuildStrategy =
+        DefaultDocsBuildStrategy.fromConfig(config);
+
+      MockedConfigReader.prototype.getString.mockReturnValue('external');
+
+      const result = await defaultDocsBuildStrategy.shouldBuild({ entity });
+
+      expect(result).toBe(false);
+    });
+  });
+});

plugins/techdocs-backend/src/service/DocsBuildStrategy.ts

@@ -0,0 +1,51 @@
+/*
+ * Copyright 2022 The Backstage Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { Entity } from '@backstage/catalog-model';
+import { Config } from '@backstage/config';
+
+/**
+ * Parameters passed to the shouldBuild method on the DocsBuildStrategy interface
+ *
+ * @public
+ */
+export type ShouldBuildParameters = {
+  entity: Entity;
+};
+
+/**
+ * A strategy for when to build TechDocs locally, and when to skip building TechDocs (allowing for an external build)
+ *
+ * @public
+ */
+export interface DocsBuildStrategy {
+  shouldBuild(params: ShouldBuildParameters): Promise<boolean>;
+}
+
+export class DefaultDocsBuildStrategy {
+  private readonly config: Config;
+
+  private constructor(config: Config) {
+    this.config = config;
+  }
+
+  static fromConfig(config: Config): DefaultDocsBuildStrategy {
+    return new DefaultDocsBuildStrategy(config);
+  }
+
+  async shouldBuild(_: ShouldBuildParameters): Promise<boolean> {
+    return this.config.getString('techdocs.builder') === 'local';
+  }
+}

plugins/techdocs-backend/src/service/index.ts

@@ -20,3 +20,7 @@ export type {
   RecommendedDeploymentOptions,
   OutOfTheBoxDeploymentOptions,
 } from './router';
+export type {
+  DocsBuildStrategy,
+  ShouldBuildParameters,
+} from './DocsBuildStrategy';

plugins/techdocs-backend/src/service/router.test.ts

@@ -38,6 +38,7 @@ import {
   RouterOptions,
 } from './router';
 import { TechDocsCache } from '../cache';
+import { DocsBuildStrategy } from './DocsBuildStrategy';
 
 jest.mock('@backstage/catalog-client');
 jest.mock('@backstage/config');
@@ -120,6 +121,9 @@ describe('createRouter', () => {
   const cache: jest.Mocked<PluginCacheManager> = {
     getClient: jest.fn(),
   };
+  const docsBuildStrategy: jest.Mocked<DocsBuildStrategy> = {
+    shouldBuild: jest.fn(),
+  };
   const outOfTheBoxOptions = {
     preparers,
     generators,
@@ -128,6 +132,7 @@ describe('createRouter', () => {
     logger: getVoidLogger(),
     discovery,
     cache,
+    docsBuildStrategy,
   };
   const recommendedOptions = {
     publisher,
@@ -135,6 +140,7 @@ describe('createRouter', () => {
     logger: getVoidLogger(),
     discovery,
     cache,
+    docsBuildStrategy,
   };
 
   beforeEach(() => {
@@ -181,10 +187,10 @@ describe('createRouter', () => {
         expect(response.status).toBe(404);
       });
 
-      it('should not check for an update without local builder', async () => {
+      it('should not check for an update when shouldBuild returns false', async () => {
         const app = await createApp(outOfTheBoxOptions);
 
-        MockedConfigReader.prototype.getString.mockReturnValue('external');
+        docsBuildStrategy.shouldBuild.mockResolvedValue(false);
         MockCachedEntityLoader.prototype.load.mockResolvedValue(entity);
         MockDocsSynchronizer.prototype.doCacheSync.mockImplementation(
           async ({ responseHandler }) =>
@@ -198,10 +204,10 @@ describe('createRouter', () => {
         expect(response.status).toBe(304);
       });
 
-      it('should error if missing builder', async () => {
+      it('should error if build is required and is missing preparer', async () => {
         const app = await createApp(recommendedOptions);
 
-        MockedConfigReader.prototype.getString.mockReturnValue('local');
+        docsBuildStrategy.shouldBuild.mockResolvedValue(true);
         MockCachedEntityLoader.prototype.load.mockResolvedValue(entity);
 
         const response = await request(app)
@@ -210,7 +216,7 @@ describe('createRouter', () => {
 
         expect(response.status).toBe(500);
         expect(response.text).toMatch(
-          /Invalid configuration\. 'techdocs\.builder' was set to 'local' but no 'preparer' was provided to the router initialization/,
+          /Invalid configuration\. docsBuildStrategy\.shouldBuild returned 'true', but no 'preparer' was provided to the router initialization./,
         );
 
         expect(MockDocsSynchronizer.prototype.doSync).toBeCalledTimes(0);
@@ -219,7 +225,7 @@ describe('createRouter', () => {
       it('should execute synchronization', async () => {
         const app = await createApp(outOfTheBoxOptions);
 
-        MockedConfigReader.prototype.getString.mockReturnValue('local');
+        docsBuildStrategy.shouldBuild.mockResolvedValue(true);
         MockCachedEntityLoader.prototype.load.mockResolvedValue(entity);
         MockDocsSynchronizer.prototype.doSync.mockImplementation(
           async ({ responseHandler }) =>
@@ -244,7 +250,7 @@ describe('createRouter', () => {
       it('should return on updated', async () => {
         const app = await createApp(outOfTheBoxOptions);
 
-        MockedConfigReader.prototype.getString.mockReturnValue('local');
+        docsBuildStrategy.shouldBuild.mockResolvedValue(true);
         MockCachedEntityLoader.prototype.load.mockResolvedValue(entity);
         MockDocsSynchronizer.prototype.doSync.mockImplementation(
           async ({ responseHandler }) => {
@@ -297,10 +303,10 @@ describe('createRouter', () => {
         expect(response.status).toBe(404);
       });
 
-      it('should not check for an update without local builder', async () => {
+      it('should not check for an update when shouldBuild returns false', async () => {
         const app = await createApp(outOfTheBoxOptions);
 
-        MockedConfigReader.prototype.getString.mockReturnValue('external');
+        docsBuildStrategy.shouldBuild.mockResolvedValue(false);
         MockCachedEntityLoader.prototype.load.mockResolvedValue(entity);
         MockDocsSynchronizer.prototype.doCacheSync.mockImplementation(
           async ({ responseHandler }) =>
@@ -322,10 +328,10 @@ data: {"updated":false}
         );
       });
 
-      it('should error if missing builder', async () => {
+      it('should error if build is required and is missing preparer', async () => {
         const app = await createApp(recommendedOptions);
 
-        MockedConfigReader.prototype.getString.mockReturnValue('local');
+        docsBuildStrategy.shouldBuild.mockResolvedValue(true);
         MockCachedEntityLoader.prototype.load.mockResolvedValue(entity);
 
         const response = await request(app)
@@ -337,7 +343,7 @@ data: {"updated":false}
         expect(response.get('content-type')).toBe('text/event-stream');
         expect(response.text).toEqual(
           `event: error
-data: "Invalid configuration. 'techdocs.builder' was set to 'local' but no 'preparer' was provided to the router initialization."
+data: "Invalid configuration. docsBuildStrategy.shouldBuild returned 'true', but no 'preparer' was provided to the router initialization."
 
 `,
         );
@@ -348,7 +354,7 @@ data: "Invalid configuration. 'techdocs.builder' was set to 'local' but no 'prep
       it('should execute synchronization', async () => {
         const app = await createApp(outOfTheBoxOptions);
 
-        MockedConfigReader.prototype.getString.mockReturnValue('local');
+        docsBuildStrategy.shouldBuild.mockResolvedValue(true);
         MockCachedEntityLoader.prototype.load.mockResolvedValue(entity);
         MockDocsSynchronizer.prototype.doSync.mockImplementation(
           async ({ responseHandler }) =>
@@ -376,7 +382,7 @@ data: "Invalid configuration. 'techdocs.builder' was set to 'local' but no 'prep
       it('should return an event-stream', async () => {
         const app = await createApp(outOfTheBoxOptions);
 
-        MockedConfigReader.prototype.getString.mockReturnValue('local');
+        docsBuildStrategy.shouldBuild.mockResolvedValue(true);
         MockCachedEntityLoader.prototype.load.mockResolvedValue(entity);
         MockDocsSynchronizer.prototype.doSync.mockImplementation(
           async ({ responseHandler }) => {

plugins/techdocs-backend/src/service/router.ts

@@ -35,6 +35,10 @@ import { ScmIntegrations } from '@backstage/integration';
 import { DocsSynchronizer, DocsSynchronizerSyncOpts } from './DocsSynchronizer';
 import { createCacheMiddleware, TechDocsCache } from '../cache';
 import { CachedEntityLoader } from './CachedEntityLoader';
+import {
+  DefaultDocsBuildStrategy,
+  DocsBuildStrategy,
+} from './DocsBuildStrategy';
 
 /**
  * Required dependencies for running TechDocs in the "out-of-the-box"
@@ -51,6 +55,7 @@ export type OutOfTheBoxDeploymentOptions = {
   database?: Knex; // TODO: Make database required when we're implementing database stuff.
   config: Config;
   cache: PluginCacheManager;
+  docsBuildStrategy?: DocsBuildStrategy;
 };
 
 /**
@@ -65,6 +70,7 @@ export type RecommendedDeploymentOptions = {
   discovery: PluginEndpointDiscovery;
   config: Config;
   cache: PluginCacheManager;
+  docsBuildStrategy?: DocsBuildStrategy;
 };
 
 /**
@@ -99,6 +105,8 @@ export async function createRouter(
   const router = Router();
   const { publisher, config, logger, discovery } = options;
   const catalogClient = new CatalogClient({ discoveryApi: discovery });
+  const docsBuildStrategy =
+    options.docsBuildStrategy ?? DefaultDocsBuildStrategy.fromConfig(config);
 
   // Entities are cached to optimize the /static/docs request path, which can be called many times
   // when loading a single techdocs page.
@@ -210,10 +218,13 @@ export async function createRouter(
       responseHandler = createEventStream(res);
     }
 
-    // techdocs-backend will only try to build documentation for an entity if techdocs.builder is set to 'local'
-    // If set to 'external', it will assume that an external process (e.g. CI/CD pipeline
-    // of the repository) is responsible for building and publishing documentation to the storage provider
-    if (config.getString('techdocs.builder') !== 'local') {
+    // By default, techdocs-backend will only try to build documentation for an entity if techdocs.builder is set to
+    // 'local'. If set to 'external', it will assume that an external process (e.g. CI/CD pipeline
+    // of the repository) is responsible for building and publishing documentation to the storage provider.
+    // Altering the implementation of the injected docsBuildStrategy allows for more complex behaviours, based on
+    // either config or the properties of the entity (e.g. annotations, labels, spec fields etc.).
+    const shouldBuild = await docsBuildStrategy.shouldBuild({ entity });
+    if (!shouldBuild) {
       // However, if caching is enabled, take the opportunity to check and
       // invalidate stale cache entries.
       if (cache) {
@@ -244,7 +255,7 @@ export async function createRouter(
 
     responseHandler.error(
       new Error(
-        "Invalid configuration. 'techdocs.builder' was set to 'local' but no 'preparer' was provided to the router initialization.",
+        "Invalid configuration. docsBuildStrategy.shouldBuild returned 'true', but no 'preparer' was provided to the router initialization.",
       ),
     );
   });