Merge branch 'master' into how-to-use-other-plugins

Signed-off-by: Patrik Oldsberg <poldsberg@gmail.com>
2023-09-26 10:47:30 +02:00
parent 3b97f2212f e5b6767025
commit ba8dee5f1c
1608 changed files with 63779 additions and 13205 deletions
@@ -28,6 +28,53 @@ The providers available as server side are:
 - `localKubectlProxy`
 - `serviceAccount`

+### AWS
+
+For AWS, in addition to the "kubernetes" configuration, you will have to set up AWS authentication. The AWS server-side authentication provider uses [AWS Identity and Access Management (IAM)][3] to authenticate to the target Account(s), you can read more about it on the page for the [Integration AWS node][4].
+
+Using the plugin, you can authenticate to several AWS accounts using either [static AWS Access keys][5] or short-lived Access keys generated by [assuming a role][6], for either case you will need to install the [AWS CLI utility][7] and set it up following the steps in the linked documentation.
+
+If you have generated static AWS security credentials, the configuration block for AWS will look like this:
+
+```yaml
+aws:
+  mainAccount:
+  accounts:
+    - accountId: '<account-number>'
+      accessKeyId: ${AWS_ACCESS_KEY_ID}
+      secretAccessKey: ${AWS_SECRET_ACCESS_KEY}
+      region: <region>
+  accountDefaults:
+```
+
+If your environment is set up to assume a role, the configuration would instead look like this:
+
+```yaml
+aws:
+  mainAccount:
+    roleName: <name-of-the-role-to-assume>
+    region: <region>
+  accounts:
+  accountDefaults:
+```
+
+Either of these sections needs to be present for the Kubernetes configuration to use the `aws` `authProvider`. The Kubernetes configuration looks like this:
+
+```yaml
+kubernetes:
+  serviceLocatorMethod:
+    type: 'multiTenant'
+  clusterLocatorMethods:
+    - type: 'config'
+      clusters:
+        - url: https://<unique-identifier>.<region>.eks.amazonaws.com
+          name: <cluster-name-to-use>
+          authProvider: 'aws'
+          caData: ${EKS_CA_DATA}
+```
+
+You get both, the cluster `url` and `caData` directly from the AWS console by going to `EKS` > `Your cluster` > `Overview` > `Details`. You will find them under 'API server endpoint' and 'Certificate authority' respectively.
+
 ### Azure

 The Azure server side authentication provider works by authenticating on the server with
@@ -72,3 +119,8 @@ The providers available as client side are:

 [1]: https://docs.microsoft.com/en-us/azure/aks/managed-aad
 [2]: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest
+[3]: https://docs.aws.amazon.com/IAM/latest/UserGuide/when-to-use-iam.html
+[4]: https://github.com/backstage/backstage/blob/master/packages/integration-aws-node/README.md
+[5]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
+[6]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html
+[7]: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
@@ -279,7 +279,65 @@ where the `items` array has _the same length_ and _the same order_ as the input

 ## Locations

-TODO
+### `GET /locations`
+
+Lists locations.
+
+Response type is JSON, on the form
+
+```json
+[
+  {
+    "data": {
+      "id": "b9784c38-7118-472f-9e22-5638fc73bab0",
+      "target": "https://git.example.com/example-project/example-repository/blob/main/catalog-info.yaml",
+      "type": "url"
+    }
+  }
+]
+```
+
+### `POST /locations`
+
+Adds a location to be ingested by the catalog.
+
+If successful the response code will be `HTTP/1.1 201 Created` and a JSON on the form
+
+```json
+{
+  "entities": [],
+  "location": {
+    "id": "b9784c38-7118-472f-9e22-5638fc73bab0",
+    "target": "https://git.example.com/example-project/example-repository/blob/main/catalog-info.yaml",
+    "type": "url"
+  }
+}
+```
+
+If the location already exists the response will be `HTTP/1.1 409 Conflict` and a JSON on the form
+
+```json
+{
+  "error": {
+    "message": "Location url:https://git.example.com/example-project/example-repository/blob/main/catalog-info.yaml already exists",
+    "name": "ConflictError",
+    "stack": "ConflictError: Location url:https://git.example.com/example-project/example-repository/blob/main/catalog-info.yaml already exists\n..."
+  },
+  "request": {
+    "method": "POST",
+    "url": "/locations"
+  },
+  "response": {
+    "statusCode": 409
+  }
+}
+```
+
+Supports the `?dryRun=true` query parameter, which will perform validation and not write anything to the database. In the event of successfully passing validation, the `entities` field of the response JSON will be populated with entities present in the location.
+
+### `DELETE /locations/<uid>`
+
+Delete a location by its id. On success response code will be `HTTP/1.1 204 No Content`.

 ## Other

@@ -123,3 +123,43 @@ source that should be mirrored into Backstage. To make Backstage a mirror of
 this remote source, users cannot also register new entities with e.g. the
 [catalog-import](https://github.com/backstage/backstage/tree/master/plugins/catalog-import)
 plugin.
+
+## Clean up orphaned entities
+
+In short entities can become orphaned through multiple means, such as when a catalog-info YAML file is moved from one place to another in the version control system without updating the registration in the catalog. For safety reasons the default behavior is to just tag the orphaned entities, and keep them around. You can read more about orphaned entities [here](life-of-an-entity.md#orphaning).
+
+However, if you do with to automatically remove the orphaned entities, you can use the following configuration, and everything with an orphaned entity tag will be eventually deleted.
+
+```yaml
+catalog:
+  orphanStrategy: delete
+```
+
+## Processing Interval
+
+The [processing loop](https://backstage.io/docs/features/software-catalog/life-of-an-entity) is
+responsible for running your registered processors on all entities, on a certain
+interval. That interval can be configured with the `processingInterval`
+app-config parameter.
+
+```yaml
+catalog:
+  processingInterval: { minutes: 45 }
+```
+
+The value is a duration object, that has one or more of the fields `years`,
+`months`, `weeks`, `days`, `hours`, `minutes`, `seconds`, and `milliseconds`.
+You can combine them, for example as `{ hours: 1, minutes: 15 }` which
+essentially means that you want the processing loop to visit entities roughly
+once every 75 minutes.
+
+Note that this is only a suggested minimum, and the actual interval may be
+longer. Internally, the catalog will scale up this number by a small factor and
+choose random numbers in that range to spread out the load. If the catalog is
+overloaded and cannot process all entities during the interval, the time taken
+between processing runs of any given entity may also be longer than specified
+here.
+
+Setting this value too low risks exhausting rate limits on external systems that
+are queried by processors, such as version control systems housing catalog-info
+files.
@@ -467,7 +467,7 @@ The next step is to create a custom processor for your new entity kind. This
 will be used within the catalog to make sure that it's able to ingest and
 validate entities of our new kind. Just like with the definition package, you
 can find inspiration in for example the existing
-[ScaffolderEntitiesProcessor](https://github.com/backstage/backstage/tree/master/plugins/scaffolder-backend/src/processor/ScaffolderEntitiesProcessor.ts).
+[ScaffolderEntitiesProcessor](https://github.com/backstage/backstage/tree/master/plugins/catalog-backend-module-scaffolder-entity-model/src/processor/ScaffolderEntitiesProcessor.ts).
 We also provide a high-level example of what a catalog process for a custom
 entity might look like:

@@ -66,10 +66,14 @@ have to supply a (unique) name, and accept a connection from the environment
 through which you can issue writes. The rest is up to the individual provider
 implementation.

-It is up to you where you put the code for this new processor class. For quick
+It is up to you where you put the code for this new provider class. For quick
 experimentation you could place it in your backend package, but we recommend
-putting all extensions like this in a backend plugin package of their own in the
-`plugins` folder of your Backstage repo.
+putting all extensions like this in a backend module package of their own in the
+`plugins` folder of your Backstage repo:
+
+```sh
+yarn new --select backend-module --option id=catalog
+```

 The class will have this basic structure:

@@ -495,8 +499,12 @@ subclass that can be added to this catalog builder.

 It is up to you where you put the code for this new processor class. For quick
 experimentation you could place it in your backend package, but we recommend
-putting all extensions like this in a backend plugin package of their own in the
-`plugins` folder of your Backstage repo.
+putting all extensions like this in a backend module package of their own in the
+`plugins` folder of your Backstage repo:
+
+```sh
+yarn new --select backend-module --option id=catalog
+```

 The class will have this basic structure:

@@ -222,7 +222,8 @@ either, it becomes _orphaned_. The end result is as follows:
 - The stitching process injects a `backstage.io/orphan: 'true'` annotation on
  the child entity.
 - The child entity is _not_ removed from the catalog, but stays around until
-  explicitly deleted via the catalog API, or "reclaimed" by the original parent
+  explicitly deleted via the catalog API, implicitly if `orphanStrategy: delete`
+  configuration is set, or until it is "reclaimed" by the original parent
  or another parent starting to reference it.
 - The catalog page in Backstage for the child entity detects the new annotation
  and informs users about the orphan status.
@@ -259,9 +260,13 @@ entities without explicit owner consent. The catalog therefore takes the stance
 that entities that often were added by direct user action should also be deleted
 only by direct user action.

-It is possible to use the catalog API to build automated "reaper" systems that
-finally delete entities that are orphaned. This is however not something that's
-provided out of the box.
+However, if you want to delete orphaned entities automatically anyway, you can
+enable the automated clean up with the following app-config option.
+
+```
+catalog:
+  orphanStrategy: delete
+```

 ## Implicit Deletion

@@ -100,6 +100,21 @@ alongside the entity's source code, the value of this annotation can point to an
 absolute URL, matching the location reference string format outlined above, for
 example: `url:https://github.com/backstage/backstage/tree/master`

+### backstage.io/techdocs-entity
+
+```yaml
+# Example:
+metadata:
+  annotations:
+    backstage.io/techdocs-entity: component:default/example
+```
+
+The value of this annotation informs of an external entity that owns the TechDocs.
+This allows you to reference TechDocs from a single source without either duplicating
+the TechDocs in the TechDocs page or needing multiple builds of the same docs.
+
+This is for situations where you have complex systems where they share a single repo, and likely a single TechDoc location.
+
 ### backstage.io/view-url, backstage.io/edit-url

 ```yaml
@@ -627,3 +627,98 @@ output things. You can grab that output using `steps.$stepId.output.$property`.
 You can read more about all the `inputs` and `outputs` defined in the actions in
 code part of the `JSONSchema`, or you can read more about our
 [built in actions](./builtin-actions.md).
+
+## Built in Filters
+
+Template filters are functions that help you transform data, extract specific information,
+and perform various operations in Scaffolder Templates.
+
+This section introduces the built-in filters provided by Backstage and offers examples of
+how to use them in the Scaffolder templates. It's important to mention that Backstage also leverages the
+native filters from the Nunjucks library. For a complete list of these native filters and their usage,
+refer to the [Nunjucks documentation](https://mozilla.github.io/nunjucks/templating.html#builtin-filters).
+
+### parseRepoUrl
+
+The `parseRepoUrl` filter parse a repository URL into
+its components, such as `owner`, repository `name`, and more.
+
+**Usage Example:**
+
+```yaml
+- id: log
+  name: Parse Repo URL
+  action: debug:log
+  input:
+    extra: ${{ parameters.repoUrl | parseRepoUrl }}
+```
+
+- **Input**: `github.com?repo=backstage&org=backstage`
+- **Output**: [RepoSpec](https://github.com/backstage/backstage/blob/v1.17.2/plugins/scaffolder-backend/src/scaffolder/actions/builtin/publish/util.ts#L39)
+
+### parseEntityRef
+
+The `parseEntityRef` filter allows you to extract different parts of
+an entity reference, such as the `kind`, `namespace`, and `name`.
+
+**Usage example**
+
+1. Without context
+
+```yaml
+- id: log
+  name: Parse Entity Reference
+  action: debug:log
+  input:
+    extra: ${{ parameters.owner | parseEntityRef }}
+```
+
+- **Input**: `group:techdocs`
+- **Output**: [CompoundEntityRef](https://github.com/backstage/backstage/blob/v1.17.2/packages/catalog-model/src/types.ts#L23)
+
+2. With context
+
+```yaml
+- id: log
+  name: Parse Entity Reference
+  action: debug:log
+  input:
+    extra: ${{ parameters.owner | parseEntityRef({ defaultKind:"group", defaultNamespace:"another-namespace" }) }}
+```
+
+- **Input**: `techdocs`
+- **Output**: [CompoundEntityRef](https://github.com/backstage/backstage/blob/v1.17.2/packages/catalog-model/src/types.ts#L23)
+
+### pick
+
+This `pick` filter allows you to select specific properties from an object.
+
+**Usage Example**
+
+```yaml
+- id: log
+  name: Pick
+  action: debug:log
+  input:
+    extra: ${{ parameters.owner | parseEntityRef | pick('name') }}
+```
+
+- **Input**: `{ kind: 'Group', namespace: 'default', name: 'techdocs' }`
+- **Output**: `techdocs`
+
+### projectSlug
+
+The `projectSlug` filter generates a project slug from a repository URL
+
+**Usage Example**
+
+```yaml
+- id: log
+  name: Project Slug
+  action: debug:log
+  input:
+    extra: ${{ parameters.repoUrl | projectSlug }}
+```
+
+- **Input**: `github.com?repo=backstage&org=backstage`
+- **Output**: `backstage/backstage`
@@ -122,12 +122,12 @@ page header, TechDocs Addons whose location is `Header` will not be rendered.
 Addons can, in principle, be provided by any plugin! To make it easier to
 discover available Addons, we've compiled a list of them here:

-| Addon                                                                                                                        | Package/Plugin                                     | Description                                                                                                                                                                                                                                                                                             |
-| ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [`<ExpandableNavigation />`](https://backstage.io/docs/reference/plugin-techdocs-module-addons-contrib.expandablenavigation) | `@backstage/plugin-techdocs-module-addons-contrib` | Allows TechDocs users to expand or collapse the entire TechDocs main navigation, and keeps the user's preferred state between documentation sites.                                                                                                                                                      |
-| [`<ReportIssue />`](https://backstage.io/docs/reference/plugin-techdocs-module-addons-contrib.reportissue)                   | `@backstage/plugin-techdocs-module-addons-contrib` | Allows TechDocs users to select a portion of text on a TechDocs page and open an issue against the repository that contains the documentation, populating the issue description with the selected text according to a configurable template.                                                            |
-| [`<TextSize />`](https://backstage.io/docs/reference/plugin-techdocs-module-addons-contrib.textsize)                         | `@backstage/plugin-techdocs-module-addons-contrib` | This TechDocs addon allows users to customize text size on documentation pages, they can select how much they want to increase or decrease the font size via slider or buttons. The default value for font size is 100% and this setting is kept in the browser's local storage whenever it is changed. |
-| [`<LightBox />`](https://backstage.io/docs/reference/plugin-techdocs-module-addons-contrib.lightbox)                         | `@backstage/plugin-techdocs-module-addons-contrib` | This TechDocs addon allows users to open images in a light-box on documentation pages, they can navigate between images if there are several on one page. The image size of the light-box image is the same as the image size on the document page.                                                     |
+| Addon                                                                                                                        | Package/Plugin                                     | Description                                                                                                                                                                                                                                                                                                                                                         |
+| ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [`<ExpandableNavigation />`](https://backstage.io/docs/reference/plugin-techdocs-module-addons-contrib.expandablenavigation) | `@backstage/plugin-techdocs-module-addons-contrib` | Allows TechDocs users to expand or collapse the entire TechDocs main navigation, and keeps the user's preferred state between documentation sites.                                                                                                                                                                                                                  |
+| [`<ReportIssue />`](https://backstage.io/docs/reference/plugin-techdocs-module-addons-contrib.reportissue)                   | `@backstage/plugin-techdocs-module-addons-contrib` | Allows TechDocs users to select a portion of text on a TechDocs page and open an issue against the repository that contains the documentation, populating the issue description with the selected text according to a configurable template.                                                                                                                        |
+| [`<TextSize />`](https://backstage.io/docs/reference/plugin-techdocs-module-addons-contrib.textsize)                         | `@backstage/plugin-techdocs-module-addons-contrib` | This TechDocs addon allows users to customize text size on documentation pages, they can select how much they want to increase or decrease the font size via slider or buttons. The default value for font size is 100% and this setting is kept in the browser's local storage whenever it is changed.                                                             |
+| [`<LightBox />`](https://backstage.io/docs/reference/plugin-techdocs-module-addons-contrib.lightbox)                         | `@backstage/plugin-techdocs-module-addons-contrib` | This TechDocs addon allows users to open images in a light-box on documentation pages, they can navigate between images if there are several on one page. The image size of the light-box image is the same as the image size on the document page. When clicking on the zoom icon it zooms the image to fit in the screen (similar to `background-size: contain`). |

 Got an Addon to contribute? Feel free to add a row above!

@@ -167,6 +167,12 @@ techdocs:
      # (Required) Azure Blob Storage Container Name
      containerName: 'techdocs-storage'

+      # (Optional) Azure blob storage connection string.
+      # Can be useful for local testing through azurite
+      # Defaults to undefined
+      # if provided, takes higher priority, 'techdocs.publisher.azureBlobStorage.credentials' will become irrelevant
+      connectionString: ''
+
      # (Required) An account name is required to write to a storage blob container.
      # https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key
      credentials:
@@ -263,22 +263,42 @@ Setting `generator.runIn` to `local` means you will have to make sure your
 environment is compatible with techdocs.

 You will have to install the `mkdocs` and `mkdocs-techdocs-core` package from
-pip, as well as `graphviz` and `plantuml` from your OS package manager (e.g.
+pip, optionally also `graphviz` and `plantuml` from your OS package manager (e.g.
 apt).

 You can do so by including the following lines right above `USER node` of your
 `Dockerfile`:

 ```Dockerfile
-RUN apt-get update && apt-get install -y python3 python3-pip
-RUN pip3 install mkdocs-techdocs-core==1.1.7
+RUN apt-get update && apt-get install -y python3 python3-pip python3-venv
+
+ENV VIRTUAL_ENV=/opt/venv
+RUN python3 -m venv $VIRTUAL_ENV
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
+
+RUN pip3 install mkdocs-techdocs-core
 ```

 Please be aware that the version requirement could change, you need to check our
 [`Dockerfile`](https://github.com/backstage/techdocs-container/blob/main/Dockerfile)
 and make sure to match with it.

-Note: We recommend Python version 3.7 or higher.
+On a Debian-based Docker container, Python packages must be either installed using
+the OS package manager or within a virtual environment (see the
+[related PEP](https://peps.python.org/pep-0668/)). Alternative is to use e.g.
+[pipx](https://pypa.github.io/pipx/) for installing Python packages in an isolated
+environment.
+
+The above Dockerfile snippet installs the latest `mkdocs-techdoc-core` package.
+Version numbers can be found in the corresponding
+[changelog](https://github.com/backstage/mkdocs-techdocs-core#changelog). In
+case you want to pin the version, use the example below:
+
+```Dockerfile
+RUN pip3 install mkdocs-techdocs-core==1.2.3
+```
+
+Note: We recommend Python version 3.11 or higher.

 > Caveat: Please install the `mkdocs-techdocs-core` package after all other
 > Python packages. The order is important to make sure we get correct version of
@@ -704,3 +704,32 @@ Then publish the image and use it in your config under the `techdocs.generator.d
 To use the plugin, it has to be listed in the `mkdocs.yaml` file. You can either add the plugin to your applicable files, or specify defaults.

 To make a mkdocs plugin available for all your TechDocs components you can either list it in the `techdocs.generator.mkdocs.defaultPlugins` [config](https://github.com/backstage/backstage/blob/master/plugins/techdocs-backend/config.d.ts#L64C14-L64C14), or use the `--defaultPlugin` [cli option](https://backstage.io/docs/features/techdocs/cli#generate-techdocs-site-from-a-documentation-project) depending on your setup.
+
+## Reference another components TechDocs
+
+In systems where you might have multiple entities for example a System with a Website and an API, when served from a Monorepo you might want to keep the TechDocs in one location in the repository.
+
+In this case you can add the `backstage.io/techdocs-entity` annotation and point to the owners `entityRef` and use its TechDocs. This allows the Subcomponents to read the parents docs, filling the TechDocs link on the `AboutCard` element and the Techdocs tab
+
+```yaml
+apiVersion: backstage.io/v1alpha1
+kind: System
+metadata:
+  name: example
+  namespace: default
+  title: Example
+  description: This is the parent entity
+  annotations:
+    backstage.io/techdocs-ref: dir:.
+
+---
+apiVersion: backstage.io/v1alpha1
+kind: Component
+metadata:
+  name: example-platfrom
+  title: Example Application Platform
+  namespace: default
+  description: This is the child entity
+  annotations:
+    backstage.io/techdocs-entity: system:default/example
+```