Merge branch 'master' into github-installations-limit

2021-07-14 08:03:08 +01:00
parent 5fa0c32bff 474cb76c88
commit a5d112ef49
3877 changed files with 39360 additions and 53613 deletions
@@ -23,18 +23,18 @@ during their entire life cycle.
 Each Utility API is tied to an `ApiRef` instance, which is a global singleton
 object without any additional state or functionality, its only purpose is to
 reference Utility APIs. `ApiRef`s are created using `createApiRef`, which is
-exported by `@backstage/core`. There are many
+exported by `@backstage/core-plugin-api`. There are many
 [predefined Utility APIs](../reference/utility-apis/README.md) defined in
-`@backstage/core`, and they're all exported with a name of the pattern
-`*ApiRef`, for example `errorApiRef`.
+`@backstage/core-plugin-api`, and they're all exported with a name of the
+pattern `*ApiRef`, for example `errorApiRef`.

 To access one of the Utility APIs inside a React component, use the `useApi`
-hook exported by `@backstage/core`, or the `withApis` HOC if you prefer class
-components. For example, the `ErrorApi` can be accessed like this:
+hook exported by `@backstage/core-plugin-api`, or the `withApis` HOC if you
+prefer class components. For example, the `ErrorApi` can be accessed like this:

 ```tsx
 import React from 'react';
-import { useApi, errorApiRef } from '@backstage/core';
+import { useApi, errorApiRef } from '@backstage/core-plugin-api';

 export const MyComponent = () => {
  const errorApi = useApi(errorApiRef);
@@ -52,9 +52,9 @@ Note that there is no explicit type given for `ErrorApi`. This is because the
 `errorApiRef` has the type embedded, and `useApi` is able to infer the type.

 Also note that consuming Utility APIs is not limited to plugins, it can be done
-from any component inside Backstage, including the ones in `@backstage/core`.
-The only requirement is that they are beneath the `AppProvider` in the react
-tree.
+from any component inside Backstage, including the ones in
+`@backstage/core-plugin-api`. The only requirement is that they are beneath the
+`AppProvider` in the react tree.

 ## Supplying APIs

@@ -71,8 +71,11 @@ For example, this is the default `ApiFactory` for the `ErrorApi`:
 createApiFactory({
  api: errorApiRef,
  deps: { alertApi: alertApiRef },
-  factory: ({ alertApi }) =>
-    new ErrorAlerter(alertApi, new ErrorApiForwarder()),
+  factory: ({ alertApi }) => {
+    const errorApi = new ErrorAlerter(alertApi, new ErrorApiForwarder());
+    UnhandledErrorForwarder.forward(errorApi, { hidden: false });
+    return errorApi;
+  },
 });
 ```

@@ -98,13 +101,13 @@ app, and the app itself.
 ### Core APIs

 Starting with the Backstage core library, it provides implementations for all of
-the core APIs. The core APIs are the ones exported by `@backstage/core`, such as
-the `errorApiRef` and `configApiRef`. You can find a full list of them
-[here](../reference/utility-apis/README.md).
+the core APIs. The core APIs are the ones exported by
+`@backstage/core-plugin-api`, such as the `errorApiRef` and `configApiRef`. You
+can find a full list of them [here](../reference/utility-apis/README.md).

 The core APIs are loaded for any app created with `createApp` from
-`@backstage/core`, which means that there is no step that needs to be taken to
-include these APIs in an app.
+`@backstage/core-plugin-api`, which means that there is no step that needs to be
+taken to include these APIs in an app.

 ### Plugin APIs

@@ -210,8 +213,9 @@ implement the `ErrorApi`, as it is checked by the type embedded in the

 Plugins are free to define their own Utility APIs. Simply define the TypeScript
 interface for the API, and create an `ApiRef` using `createApiRef` exported from
-`@backstage/core`. Also be sure to provide at least one implementation of the
-API, and to declare a default factory for the API in `createPlugin`.
+`@backstage/core-plugin-api`. Also be sure to provide at least one
+implementation of the API, and to declare a default factory for the API in
+`createPlugin`.

 Custom Utility APIs can be either public or private, which is up to the plugin
 to choose. Private APIs do not expose an external API surface, and it's
@@ -6,8 +6,8 @@ description: Architecture Decision Record (ADR) log on Module Export Structure

 ## Context

-With a growing number of exports of packages like `@backstage/core`, it is
-becoming more and more difficult to answer questions such as
+With a growing number of exports of packages like `@backstage/core-components`,
+it is becoming more and more difficult to answer questions such as

 > Is the export in this module also exported by the package?

@@ -86,7 +86,7 @@ import { helperFunc } from '../../lib/UtilityX/helper';
 ## Consequences

 We will actively work to rework the export structure in our codebase,
-prioritizing the library packages such as `@backstage/core` and
+prioritizing the library packages such as `@backstage/core-components` and
 `@backstage/backend-common`.

 If possible, we will add tools, such as lint rules, to help enforce the export
@@ -35,6 +35,8 @@ example `catalog` or `techdocs`):

 - `x`: Contains the main frontend code of the plugin.
 - `x-backend`: Contains the main backend code of the plugin.
+- `x-backend-module-<name>`: Contains optional modules related to the backend
+  plugin.
 - `x-react`: Contains shared widgets, hooks and similar that both the plugin
  itself (`x`) and third-party frontend plugins can depend on.
 - `x-node`: Contains utilities for backends that both the plugin backend itself
@@ -61,6 +63,10 @@ We will actively migrate existing packages that are part of a plugin to the
  `plugins/catalog-common` we might want to do an exception here, as it's a very
  central package.

+We will actively migrate optional features of backend plugins into separate
+`x-backend-module-<name>` packages, for example the more specialized processors
+in the catalog backend.
+
 The limited set of rules might not be sufficient in the future. If additional
 packages are required, we will revisit this decision and extend the pattern.

@@ -0,0 +1,150 @@
+---
+id: identity-resolver
+title: Identity resolver
+description: Identity resolvers of Backstage users after they sign-in
+---
+
+This guide explains how the identity of a Backstage user is stored inside their
+Backstage Identity Token and how you can customize the Sign-In resolvers to
+include identity and group membership information of the user from other
+external systems. This ultimately helps with determining the ownership of a
+Backstage entity by a user. The ideas here were originally proposed in the RFC
+[#4089](https://github.com/backstage/backstage/issues/4089).
+
+When a user signs in to Backstage, inside the `claims` field of their Backstage
+Token (which are standard JWT tokens) a special `ent` claim is set. `ent`
+contains a list of
+[entity references](../features/software-catalog/references.md), each of which
+denotes an identity or a membership that is relevant to the user. There is no
+guarantee that these correspond to actual existing catalog entities.
+
+Let's take an example sign-in resolver for the Google auth provider and explore
+how the `ent` field inside `claims` can be set.
+
+Inside your `packages/backend/src/plugins/auth.ts` file, you can provide custom
+sign-in resolvers and set them for any of the Authentication providers inside
+`providerFactories` of the `createRouter` imported from the
+`@backstage/plugin-auth-backend` plugin.
+
+```ts
+export default async function createPlugin({
+  ...
+}: PluginEnvironment): Promise<Router> {
+  return await createRouter({
+    ...
+    providerFactories: {
+      google: createGoogleProvider({
+        signIn: {
+          resolver: async ({ profile: { email } }, ctx) => {
+            // Call a custom validator function that checks that the email is
+            // valid and on our own company's domain, and throws an Error if it
+            // isn't
+            validateEmail(email);
+
+            // List of entity references that denote the identity and
+            // membership of the user
+            const ent = [];
+
+            // Let's use the username in the email ID as the user's default
+            // unique identifier inside Backstage.
+            const [id] = email.split('@');
+            ent.push(`User:default/${id}`)
+
+            // Let's call the internal LDAP provider to get a list of groups
+            // that the user belongs to, and add those to the list as well
+            const ldapGroups = await getLdapGroups(email);
+            ldapGroups.forEach(group => ent.push(`Group:default/${group}`))
+
+            // Issue the token containing the entity claims
+            const token = await ctx.tokenIssuer.issueToken({
+              claims: { sub: id, ent },
+            });
+            return { id, token };
+          },
+        },
+      }),
+    },
+  });
+}
+```
+
+As you can see, the generated Backstage Token now contains all the claims about
+the identity and membership of the user. Once the sign-in process is complete,
+and we need to find out if a user owns an Entity in the Software Catalog, these
+`ent` claims can be used to determine the ownership.
+
+According to the RFC, the definition of the ownership of an entity E, for a user
+U, is as follows:
+
+- Get all the `ownedBy` relations of E, and call them O
+- Get all the claims of the user U and call them C
+- If any C matches any O, return `true`
+- Get all Group entities that U is a member of, using the regular
+  `memberOf`/`hasMember` relation mechanism, and call them G
+- If any G matches any O, return `true`
+- Otherwise, return `false`
+
+## Default sign-in resolvers
+
+Of course you don't have to customize the sign-in resolver if you don't need to.
+The Auth backend plugin comes with a set of default sign-in resolvers which you
+can use. For example - the Google provider has a default email-based sign-in
+resolver, which will search the catalog for a single user entity that has a
+matching `google.com/email` annotation.
+
+It can be enabled like this
+
+```tsx
+// File: packages/backend/src/plugins/auth.ts
+import { googleEmailSignInResolver } from '@backstage/plugin-auth-backend';
+
+export default async function createPlugin({
+  ...
+}: PluginEnvironment): Promise<Router> {
+  return await createRouter({
+    ...
+    providerFactories: {
+      google: createGoogleProvider({
+        signIn: {
+          resolver: googleEmailSignInResolver
+        }
+...
+```
+
+## AuthHandler
+
+Similar to a custom sign-in resolver, you can also write a custom auth handler
+function which is used to verify and convert the auth response into the profile
+that will be presented to the user. This is where you can customize things like
+display name and profile picture.
+
+This is also the place where you can do authorization and validation of the user
+and throw errors if the user should not be allowed access in Backstage.
+
+```tsx
+// File: packages/backend/src/plugins/auth.ts
+export default async function createPlugin({
+  ...
+}: PluginEnvironment): Promise<Router> {
+  return await createRouter({
+    ...
+    providerFactories: {
+      google: createGoogleProvider({
+        authHandler: async ({
+          fullProfile  // Type: passport.Profile,
+          idToken      // Type: (Optional) string,
+        }) => {
+          // Custom validation code goes here
+          return {
+            profile: {
+              email,
+              picture,
+              displayName,
+            }
+          };
+        }
+      })
+    }
+  })
+}
+```
@@ -66,7 +66,8 @@ built-in providers:

 ```diff
 # packages/app/src/App.tsx
-+ import { githubAuthApiRef, SignInProviderConfig, SignInPage } from '@backstage/core';
+ import { githubAuthApiRef } from '@backstage/core-plugin-api';
+ import { SignInProviderConfig, SignInPage } from '@backstage/core-components';

 + const githubProvider: SignInProviderConfig = {
 +  id: 'github-auth-provider',
@@ -14,7 +14,7 @@ to various third party APIs.
 There are occasions when the user wants to perform actions towards third party
 services that require authorization via OAuth. Backstage provides standardized
 [Utility APIs](../api/utility-apis.md) such as the
-[GoogleAuthApi](https://github.com/backstage/backstage/blob/master/packages/core-api/src/apis/definitions/auth.ts)
+[GoogleAuthApi](https://github.com/backstage/backstage/blob/master/packages/core-plugin-api/src/apis/definitions/auth.ts)
 for that use-case. Backstage also includes a set of implementations of these
 APIs that integrate with the
 [auth-backend](https://github.com/backstage/backstage/tree/master/plugins/auth-backend)
@@ -38,7 +38,7 @@ choose an account to log in with, and accept or reject the request. If the user
 accepts the login request, a token is issued, and any holder of the token can
 use it to make authenticated requests towards the third party service.

-## OAuth in @backstage/core-api and auth-backend
+## OAuth in @backstage/core-app-api and auth-backend

 The default OAuth implementation in Backstage is based on an OAuth server-side
 offline access flow, which means that it uses the backend as a helper in order
@@ -59,8 +59,8 @@ easier to make authenticated requests inside a plugin.
 The following describes the OAuth flow implemented by the
 [auth-backend](https://github.com/backstage/backstage/tree/master/plugins/auth-backend)
 and
-[DefaultAuthConnector](https://github.com/backstage/backstage/blob/master/packages/core-api/src/lib/AuthConnector/DefaultAuthConnector.ts)
-in `@backstage/core-api`.
+[DefaultAuthConnector](https://github.com/backstage/backstage/blob/master/packages/core-app-api/src/lib/AuthConnector/DefaultAuthConnector.ts)
+in `@backstage/core-app-api`.

 Component and APIs can request Access or ID Tokens from any available Auth
 provider. If there already exists a cached fresh token that covers (at least)
@@ -45,8 +45,8 @@ pieces in place that can be used.
 #### Identity for Plugin Developers

 As a plugin developer, there are two main touchpoints for identities: the
-`IdentityApi` exported by `@backstage/core` via the `identityApiRef`, and a not
-yet existing middleware exported by `@backstage/backend-common`.
+`IdentityApi` exported by `@backstage/core-plugin-api` via the `identityApiRef`,
+and a not yet existing middleware exported by `@backstage/backend-common`.

 The `IdentityApi` gives access to the signed-in user's identity in the frontend.
 It provides access to the user's ID, lightweight profile information, and an ID
@@ -61,8 +61,9 @@ https://github.com/backstage/backstage/issues/1435.

 If you're setting up your own Backstage app, or want to add a new identity
 provider, there are three touchpoints: the frontend auth APIs in
-`@backstage/core-api`, the backend auth providers in `auth-backend`, and the
-`SignInPage` component configured in the Backstage app via `createApp`.
+`@backstage/core-app-api` and `@backstage/core-plugin-api`, the backend auth
+providers in `auth-backend`, and the `SignInPage` component configured in the
+Backstage app via `createApp`.

 The frontend APIs and backend providers are tightly coupled together for each
 auth provider, and together they implement an e2e auth flow. Only some auth
@@ -81,10 +82,10 @@ The final piece of the puzzle is the `SignInPage` component that can be
 configured as part of the app. Without a sign-in page, Backstage will fall back
 to a `guest` identity for all users, without any ID token. To enable sign-in, a
 `SignInPage` needs to be configured, which in turn has to supply a user to the
-app. The `@backstage/core` package provides a basic sign-in page that allows
-both the user and the app developer to choose between a couple of different
-sign-in methods, or to designate a single provider that may also be logged in to
-automatically.
+app. The `@backstage/core-components` package provides a basic sign-in page that
+allows both the user and the app developer to choose between a couple of
+different sign-in methods, or to designate a single provider that may also be
+logged in to automatically.

 ## Further Reading

@@ -554,9 +554,7 @@ Options:
 Scope: `root`

 Validate `@backstage` dependencies within the repo, making sure that there are
-no duplicates of packages that might lead to breakages. For example,
-`@backstage/core` must not be loaded in twice, so having two different versions
-of it installed will cause this command to exit with an error.
+no duplicates of packages that might lead to breakages.

 By supplying the `--fix` flag the command will attempt to fix any conflict that
 can be resolved by editing `yarn.lock`, but will not attempt to search for
@@ -93,6 +93,21 @@ declare the visibility of a leaf node of `type: "string"`.
 | `backend`    | (Default) Only in backend                                          |
 | `secret`     | Only in backend and may be excluded from logs for security reasons |

+You can set visibility with an `@visibility` comment in the `Config` Typescript
+interface.
+
+```ts
+export interface Config {
+  app: {
+    /**
+     * Frontend root URL
+     * @visibility frontend
+     */
+    baseUrl: string;
+  };
+}
+```
+
 ## Validation

 Schemas can be validated using the `backstage-cli config:check` command. If you
@@ -112,7 +112,7 @@ example `getString`. These will throw an error if there is no value available.

 The [ConfigApi](../reference/utility-apis/Config.md) in the frontend is a
 [UtilityApi](../api/utility-apis.md). It's accessible as usual via the
-`configApiRef` exported from `@backstage/core`.
+`configApiRef` exported from `@backstage/core-plugin-api`.

 Depending on the config api in another API is slightly different though, as the
 `ConfigApi` implementation is supplied via the App itself and not instantiated
@@ -123,7 +123,7 @@ for an example of how this wiring is done.
 For standalone plugin setups in `dev/index.ts`, register a factory with a
 statically mocked implementation of the config API. Use the `ConfigReader` from
 `@backstage/config` to create an instance and register it for the `configApiRef`
-from `@backstage/core`.
+from `@backstage/core-plugin-api`.

 ## Accessing ConfigApi in Backend Plugins

@@ -59,16 +59,17 @@ Once the host build is complete, we are ready to build our image. The following
 FROM node:14-buster-slim

 WORKDIR /app
-
 # Copy repo skeleton first, to avoid unnecessary docker cache invalidation.
 # The skeleton contains the package.json of each package in the monorepo,
 # and along with yarn.lock and the root package.json, that's enough to run yarn install.
-ADD yarn.lock package.json packages/backend/dist/skeleton.tar.gz ./
+COPY yarn.lock package.json packages/backend/dist/skeleton.tar.gz ./
+RUN tar xzf skeleton.tar.gz && rm skeleton.tar.gz

 RUN yarn install --frozen-lockfile --production --network-timeout 300000 && rm -rf "$(yarn cache dir)"

 # Then copy the rest of the backend bundle, along with any other files we might want.
-ADD packages/backend/dist/bundle.tar.gz app-config.yaml ./
+COPY packages/backend/dist/bundle.tar.gz app-config.yaml ./
+RUN tar xzf bundle.tar.gz && rm bundle.tar.gz

 CMD ["node", "packages/backend", "--config", "app-config.yaml"]
 ```
@@ -25,7 +25,7 @@ component, which are then displayed both visually and with sample code to be
 copied.

 When custom Backstage components are created, they are placed in the
-`@backstage/core` package and added to the Storybook.
+`@backstage/core-components` package and added to the Storybook.

 There may be times where an existing Material-UI component (in
 `@material-ui/core`) is sufficient and doesn't need to be wrapped or duplicated.
@@ -68,7 +68,7 @@ The base URL to the Kubernetes control plane. Can be found by using the
 ##### `clusters.\*.name`

 A name to represent this cluster, this must be unique within the `clusters`
-array. Users will see this value in the Service Catalog Kubernetes plugin.
+array. Users will see this value in the Software Catalog Kubernetes plugin.

 ##### `clusters.\*.authProvider`

@@ -195,7 +195,7 @@ annotations:

 #### Labeling Kubernetes components

-In order for Kubernetes components to show up in the service catalog as a part
+In order for Kubernetes components to show up in the software catalog as a part
 of an entity, Kubernetes components themselves can have the following label:

 ```yaml
@@ -2,7 +2,7 @@
 id: overview
 title: Kubernetes
 sidebar_label: Overview
-description: Monitoring Kubernetes based services with the service catalog
+description: Monitoring Kubernetes based services with the software catalog
 ---

 Kubernetes in Backstage is a tool that's designed around the needs of service
@@ -29,7 +29,7 @@ Backstage app with the following contents:

 ```tsx
 import React from 'react';
-import { Content, Header, Page } from '@backstage/core';
+import { Content, Header, Page } from '@backstage/core-components';
 import { Grid, List, Card, CardContent } from '@material-ui/core';
 import {
  SearchBar,
@@ -142,7 +142,6 @@ export default async function createPlugin({
  const indexBuilder = new IndexBuilder({ logger, searchEngine });

  indexBuilder.addCollator({
-    type: 'software-catalog',
    defaultRefreshIntervalSeconds: 600,
    collator: new DefaultCatalogCollator({ discovery }),
  });
@@ -262,21 +261,21 @@ const indexBuilder = new IndexBuilder({ logger, searchEngine });
 ```

 Backstage Search can be used to power search of anything! Plugins like the
-Catalog offer default [collators](./concepts.md#collators) which are responsible
-for providing documents [to be indexed](./concepts.md#documents-and-indices).
-You can register any number of collators with the `IndexBuilder` like this:
+Catalog offer default [collators](./concepts.md#collators) (e.g.
+[DefaultCatalogCollator](https://github.com/backstage/backstage/blob/df12cc25aa4934a98bc42ed03c07f64a1a0a9d72/plugins/catalog-backend/src/search/DefaultCatalogCollator.ts))
+which are responsible for providing documents
+[to be indexed](./concepts.md#documents-and-indices). You can register any
+number of collators with the `IndexBuilder` like this:

 ```typescript
 const indexBuilder = new IndexBuilder({ logger, searchEngine });

 indexBuilder.addCollator({
-  type: 'software-catalog',
  defaultRefreshIntervalSeconds: 600,
  collator: new DefaultCatalogCollator({ discovery }),
 });

 indexBuilder.addCollator({
-  type: 'my-custom-stuff',
  defaultRefreshIntervalSeconds: 3600,
  collator: new MyCustomCollator(),
 });
@@ -290,7 +289,6 @@ its `defaultRefreshIntervalSeconds` value, like this:

 ```typescript {3}
 indexBuilder.addCollator({
-  type: 'software-catalog',
  defaultRefreshIntervalSeconds: 600,
  collator: new DefaultCatalogCollator({ discovery }),
 });
@@ -654,7 +654,7 @@ spec:
  steps:
    - id: fetch-base
      name: Fetch Base
-      action: fetch:cookiecutter
+      action: fetch:template
      input:
        url: ./template
        values:
@@ -1,29 +1,29 @@
 ---
 id: software-catalog-overview
-title: Backstage Service Catalog (alpha)
+title: Backstage Software Catalog (alpha)
 sidebar_label: Overview
 # prettier-ignore
-description: The Backstage Service Catalog — actually, a software catalog, since it includes more than just services
+description: The Backstage Software Catalog
 ---

-## What is a Service Catalog?
+## What is a Software Catalog?

-The Backstage Service Catalog — actually, a software catalog, since it includes
+The Backstage Software Catalog — actually, a software catalog, since it includes
 more than just services — is a centralized system that keeps track of ownership
 and metadata for all the software in your ecosystem (services, websites,
 libraries, data pipelines, etc). The catalog is built around the concept of
 [metadata YAML files](descriptor-format.md) stored together with the code, which
 are then harvested and visualized in Backstage.

-![service-catalog](https://backstage.io/blog/assets/6/header.png)
+![software-catalog](https://backstage.io/blog/assets/6/header.png)

 ## How it works

-Backstage and the Backstage Service Catalog make it easy for one team to manage
+Backstage and the Backstage Software Catalog make it easy for one team to manage
 10 services — and makes it possible for your company to manage thousands of
 them.

-More specifically, the Service Catalog enables two main use-cases:
+More specifically, the Software Catalog enables two main use-cases:

 1. Helping teams manage and maintain the software they own. Teams get a uniform
   view of all their software; services, libraries, websites, ML models — you
@@ -34,15 +34,14 @@ More specifically, the Service Catalog enables two main use-cases:
 ## Getting Started

 The Software Catalog is available to browse at `/catalog`. If you've followed
-[Installing in your Backstage App](./installation.md) in your separate App or
-[Getting Started with Backstage](../../getting-started) for this repo, you
-should be able to browse the catalog at `http://localhost:3000`.
+[Getting Started with Backstage](../../getting-started), you should be able to
+browse the catalog at `http://localhost:3000`.

-![](../../assets/software-catalog/service-catalog-home.png)
+![](../../assets/software-catalog/software-catalog-home.png)

 ## Adding components to the catalog

-The source of truth for the components in your service catalog are
+The source of truth for the components in your software catalog are
 [metadata YAML files](descriptor-format.md) stored in source control (GitHub,
 GitHub Enterprise, GitLab, ...).

@@ -105,11 +104,11 @@ them, and do so using their normal Git workflow.
 ![](../../assets/software-catalog/bsc-edit.png)

 Once the change has been merged, Backstage will automatically show the updated
-metadata in the service catalog after a short while.
+metadata in the software catalog after a short while.

 ## Finding software in the catalog

-By default the service catalog shows components owned by the team of the logged
+By default the software catalog shows components owned by the team of the logged
 in user. But you can also switch to _All_ to see all the components across your
 company's software ecosystem. Basic inline _search_ and _column filtering_ makes
 it easy to browse a big set of components.
@@ -125,7 +124,7 @@ _starring_ of components:

 ## Integrated tooling through plugins

-The service catalog is a great way to organize the infrastructure tools you use
+The software catalog is a great way to organize the infrastructure tools you use
 to manage the software. This is how Backstage creates one developer portal for
 all your tools. Rather than asking teams to jump between different
 infrastructure UIs (and incurring additional cognitive overhead each time they
@@ -1,177 +0,0 @@
---
-id: installation
-title: Installing in your Backstage App
-description: Documentation on How to install Backstage Plugin
---
-
-The catalog plugin comes in two packages, `@backstage/plugin-catalog` and
-`@backstage/plugin-catalog-backend`. Each has their own installation steps,
-outlined below.
-
-## Installing @backstage/plugin-catalog
-
-> **Note that if you used `npx @backstage/create-app`, the plugin is already
-> installed and you can skip to
-> [adding entries to the catalog](#adding-entries-to-the-catalog)**
-
-The catalog frontend plugin should be installed in your `app` package, which is
-created as a part of `@backstage/create-app`. To install the package, run:
-
-```bash
-# From your Backstage root directory
-cd packages/app
-yarn add @backstage/plugin-catalog
-```
-
-### Adding the Plugin to your `packages/app`
-
-Add the two pages that the catalog plugin provides to your app. You can choose
-any name for these routes, but we recommend the following:
-
-```tsx
-// packages/app/src/App.tsx
-import {
-  catalogPlugin,
-  CatalogIndexPage,
-  CatalogEntityPage,
-} from '@backstage/plugin-catalog';
-
-// Add to the top-level routes, directly within <FlatRoutes>
-<Route path="/catalog" element={<CatalogIndexPage />} />
-<Route path="/catalog/:namespace/:kind/:name" element={<CatalogEntityPage />}>
-  {/*
-    This is the root of the custom entity pages for your app, refer to the example app
-    in the main repo or the output of @backstage/create-app for an example
-  */}
-  <EntityPage />
-</Route>
-```
-
-The catalog plugin also has one external route that needs to be bound for it to
-function: the `createComponent` route which should link to the page where the
-user can create components. In a typical setup the create component route will
-be linked to the Scaffolder plugin's template index page:
-
-```ts
-// packages/app/src/App.tsx
-import { catalogPlugin } from '@backstage/plugin-catalog';
-import { scaffolderPlugin } from '@backstage/plugin-scaffolder';
-
-const app = createApp({
-  // ...
-  bindRoutes({ bind }) {
-    bind(catalogPlugin.externalRoutes, {
-      createComponent: scaffolderPlugin.routes.root,
-    });
-  },
-});
-```
-
-You may also want to add a link to the catalog index page to your sidebar:
-
-```tsx
-// packages/app/src/components/Root.tsx
-import HomeIcon from '@material-ui/icons/Home';
-
-// Somewhere within the <Sidebar>
-<SidebarItem icon={HomeIcon} to="/catalog" text="Home" />;
-```
-
-This is all that is needed for the frontend part of the Catalog plugin to work!
-
-## Gotchas that we will fix
-
-Since the catalog plugin currently ships with a sentry plugin `InfoCard`
-installed by default, you'll need to set `sentry.organization` in your
-`app-config.yaml`. For example:
-
-```yaml
-sentry:
-  organization: Acme Corporation
-```
-
-If you've created an app with an older version of `@backstage/create-app` or
-`@backstage/cli create-app`, be sure to remove the Welcome plugin from the app,
-as that will conflict with the catalog routes.
-
-## Installing @backstage/plugin-catalog-backend
-
-> **Note that if you used `npx @backstage/create-app`, the plugin is already
-> installed and you can skip to
-> [adding entries to the catalog](#adding-entries-to-the-catalog)**
-
-The catalog backend should be installed in your `backend` package, which is
-created as a part of `@backstage/create-app`. To install the package, run:
-
-```bash
-# From your Backstage root directory
-cd packages/backend
-yarn add @backstage/plugin-catalog-backend
-```
-
-### Adding the Plugin to your `packages/backend`
-
-You'll need to add the plugin to the `backend`'s router. You can do this by
-creating a file called `packages/backend/src/plugins/catalog.ts` with contents
-matching
-[catalog.ts in the create-app template](https://github.com/backstage/backstage/blob/master/packages/create-app/templates/default-app/packages/backend/src/plugins/catalog.ts).
-
-Once the `catalog.ts` router setup file is in place, add the router to
-`packages/backend/src/index.ts`:
-
-```ts
-import catalog from './plugins/catalog';
-
-const catalogEnv = useHotMemoize(module, () => createEnv('catalog'));
-
-const apiRouter = Router();
-/** several different routers */
-apiRouter.use('/catalog', await catalog(catalogEnv));
-```
-
-### Adding Entries to the Catalog
-
-At this point the catalog backend is installed in your backend package, but you
-will not have any entities loaded.
-
-To get up and running and try out some templates quickly, you can add some of
-our example templates through static configuration. Add the following to the
-`catalog.locations` section in your `app-config.yaml`:
-
-```yaml
-catalog:
-  locations:
-    # Backstage Example Components
-    - type: url
-      target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/components/artist-lookup-component.yaml
-    - type: url
-      target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/components/playback-order-component.yaml
-    - type: url
-      target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/components/podcast-api-component.yaml
-    - type: url
-      target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/components/queue-proxy-component.yaml
-    - type: url
-      target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/components/searcher-component.yaml
-    - type: url
-      target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/components/playback-lib-component.yaml
-    - type: url
-      target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/components/www-artist-component.yaml
-    - type: url
-      target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/components/shuffle-api-component.yaml
-```
-
-### Running the Backend
-
-Finally, start up Backstage with the new configuration:
-
-```bash
-# Run from the root to start both backend and frontend
-yarn dev
-
-# Alternatively, run only the backend from its own package
-cd packages/backend
-yarn start
-```
-
-If you've also set up the frontend plugin, you should be ready to go browse the
-catalog at [localhost:3000](http://localhost:3000) now!
@@ -51,7 +51,7 @@ spec:
  steps:
    - id: fetch-base
      name: Fetch Base
-      action: fetch:cookiecutter
+      action: fetch:template
      input:
        url: ./template
        values:
@@ -83,7 +83,7 @@ spec:
 [Template Entity](../software-catalog/descriptor-format.md#kind-template)
 contains more information about the required fields.

-Once we have a `template.yaml` ready, we can then add it to the service catalog
+Once we have a `template.yaml` ready, we can then add it to the software catalog
 for use by the scaffolder.

 You can add the template files to the catalog through
@@ -14,3 +14,54 @@ Azure, GitLab and Bitbucket.
 A list of all registered actions can be found under `/create/actions`. For local
 development you should be able to reach them at
 `http://localhost:3000/create/actions`.
+
+### Migrating from `fetch:cookiecutter` to `fetch:template`
+
+The `fetch:template` action is a new action with a similar API to
+`fetch:cookiecutter` but no dependency on `cookiecutter`. There are two options
+for migrating templates that use `fetch:cookiecutter` to use `fetch:template`:
+
+#### Using `cookiecutterCompat` mode
+
+The new `fetch:template` action has a `cookiecutterCompat` flag which should
+allow most templates built for `fetch:cookiecutter` to work without any changes.
+
+1. Update action name in `template.yaml`. The name should be changed from
+   `fetch:cookiecutter` to `fetch:template`.
+2. Set `cookiecutterCompat` to `true` in the `fetch:template` step input in
+   `template.yaml`.
+
+```diff
+  steps:
+    - id: fetch-base
+      name: Fetch Base
+-     action: fetch:cookiecutter
+     action: fetch:template
+      input:
+        url: ./skeleton
+       cookiecutterCompat: true
+        values:
+```
+
+#### Manual migration
+
+If you prefer, you can manually migrate your templates to avoid the need for
+enabling cookiecutter compatibility mode, which will result in slightly less
+verbose template variables expressions.
+
+1. Update action name in `template.yaml`. The name should be changed from
+   `fetch:cookiecutter` to `fetch:template`.
+2. Update variable syntax in file names and content. `fetch:cookiecutter`
+   expects variables to be enclosed in `{{` `}}` and prefixed with
+   `cookiecutter.`, while `fetch:template` expects variables to be enclosed in
+   `${{` `}}` and prefixed with `values.`. For example, a reference to variable
+   `myInputVariable` would need to be migrated from
+   `{{ cookiecutter.myInputVariable }}` to `${{ values.myInputVariable }}`.
+3. Replace uses of `jsonify` with `dump`. The
+   [`jsonify` filter](https://cookiecutter.readthedocs.io/en/latest/advanced/template_extensions.html#jsonify-extension)
+   is built in to `cookiecutter`, and is not available by default when using
+   `fetch:template`. The
+   [`dump` filter](https://mozilla.github.io/nunjucks/templating.html#dump) is
+   the equivalent filter in nunjucks, so an expression like
+   `{{ cookiecutter.myAwesomeList | jsonify }}` should be migrated to
+   `${{ values.myAwesomeList | dump }}`.
@@ -0,0 +1,52 @@
+---
+id: configuration
+title: Software Template Configuration
+sidebar_label: Configuration
+description: Configuration options for Backstage Software Templates
+---
+
+Backstage software templates create source code, so your Backstage application
+needs to be set up to allow repository creation.
+
+This is done in your `app-config.yaml` by adding
+[Backstage integrations](https://backstage.io/docs/integrations/) for the
+appropriate source code repository for your organization.
+
+> Note: Integrations may already be set up as part of your `app-config.yaml`.
+
+The next step is to add
+[add templates](http://backstage.io/docs/features/software-templates/adding-templates)
+to your Backstage app.
+
+### GitHub
+
+For GitHub, you can configure who can see the new repositories that are created
+by specifying `visibility` option. Valid options are `public`, `private` and
+`internal`. The `internal` option is for GitHub Enterprise clients, which means
+public within the enterprise.
+
+```yaml
+scaffolder:
+  github:
+    visibility: public # or 'internal' or 'private'
+```
+
+### Disabling Docker in Docker situation (Optional)
+
+Software Templates use
+[Cookiecutter](https://github.com/cookiecutter/cookiecutter) as a templating
+library. By default it will use the
+[scaffolder-backend/Cookiecutter](https://github.com/backstage/backstage/blob/master/plugins/scaffolder-backend/scripts/Cookiecutter.dockerfile)
+docker image.
+
+If you are running Backstage from a Docker container and you want to avoid
+calling a container inside a container, you can set up Cookiecutter in your own
+image, this will use the local installation instead.
+
+You can do so by including the following lines in the last step of your
+`Dockerfile`:
+
+```Dockerfile
+RUN apt-get update && apt-get install -y python3 python3-pip
+RUN pip3 install cookiecutter
+```
@@ -17,10 +17,8 @@ locations like GitHub or GitLab.

 ### Getting Started

-> Be sure to have covered [Installing in your Backstage App](./installation.md)
-> for your separate App or
-> [Getting Started with Backstage](../../getting-started) for this repo before
-> proceeding.
+> Be sure to have covered
+> [Getting Started with Backstage](../../getting-started) before proceeding.

 The Software Templates are available under `/create`. For local development you
 should be able to reach them at `http://localhost:3000/create`.
@@ -1,280 +0,0 @@
---
-id: installation
-title: Installing in your Backstage App
-description: Documentation on How to install Backstage App
---
-
-The scaffolder plugin comes in two packages, `@backstage/plugin-scaffolder` and
-`@backstage/plugin-scaffolder-backend`. Each has their own installation steps,
-outlined below.
-
-The Scaffolder plugin also depends on the Software Catalog. Instructions for how
-to set that up can be found [here](../software-catalog/installation.md).
-
-## Installing @backstage/plugin-scaffolder
-
-> **Note that if you used `npx @backstage/create-app`, the plugin may already be
-> present**
-
-The scaffolder frontend plugin should be installed in your `app` package, which
-is created as a part of `@backstage/create-app`. To install the package, run:
-
-```bash
-# From your Backstage root directory
-cd packages/app
-yarn add @backstage/plugin-scaffolder
-```
-
-### Adding the Plugin to your `packages/app`
-
-Add the root page that the Scaffolder plugin provides to your app. You can
-choose any path for the route, but we recommend the following:
-
-```tsx
-import { ScaffolderPage } from '@backstage/plugin-scaffolder';
-
-// Add to the top-level routes, directly within <FlatRoutes>
-<Route path="/create" element={<ScaffolderPage />} />;
-```
-
-You may also want to add a link to the template index page to your sidebar:
-
-```tsx
-import CreateComponentIcon from '@material-ui/icons/AddCircleOutline';
-
-// Somewhere within the <Sidebar>
-<SidebarItem icon={CreateComponentIcon} to="create" text="Create..." />;
-```
-
-This is all that is needed for the frontend part of the Scaffolder plugin to
-work!
-
-## Installing @backstage/plugin-scaffolder-backend
-
-> **Note that if you used `npx @backstage/create-app`, the plugin may already be
-> present**
-
-The scaffolder backend should be installed in your `backend` package, which is
-created as a part of `@backstage/create-app`. To install the package, run:
-
-```bash
-# From your Backstage root directory
-cd packages/backend
-yarn add @backstage/plugin-scaffolder-backend
-```
-
-### Adding the Plugin to your `packages/backend`
-
-You'll need to add the plugin to the `backend`'s router. You can do this by
-creating a file called `packages/backend/src/plugins/scaffolder.ts` with the
-following contents to get you up and running quickly.
-
-```ts
-import {
-  DockerContainerRunner,
-  SingleHostDiscovery,
-} from '@backstage/backend-common';
-import {
-  CookieCutter,
-  createRouter,
-  Preparers,
-  Publishers,
-  CreateReactAppTemplater,
-  Templaters,
-} from '@backstage/plugin-scaffolder-backend';
-import type { PluginEnvironment } from '../types';
-import Docker from 'dockerode';
-import { CatalogClient } from '@backstage/catalog-client';
-
-export default async function createPlugin({
-  logger,
-  config,
-  database,
-  reader,
-}: PluginEnvironment) {
-  const dockerClient = new Docker();
-  const containerRunner = new DockerContainerRunner({ dockerClient });
-
-  const cookiecutterTemplater = new CookieCutter({ containerRunner });
-  const craTemplater = new CreateReactAppTemplater({ containerRunner });
-  const templaters = new Templaters();
-
-  templaters.register('cookiecutter', cookiecutterTemplater);
-  templaters.register('cra', craTemplater);
-
-  const preparers = await Preparers.fromConfig(config, { logger });
-  const publishers = await Publishers.fromConfig(config, { logger });
-
-  const discovery = SingleHostDiscovery.fromConfig(config);
-  const catalogClient = new CatalogClient({ discoveryApi: discovery });
-
-  return await createRouter({
-    preparers,
-    templaters,
-    publishers,
-    logger,
-    config,
-    database,
-    catalogClient,
-    reader,
-  });
-}
-```
-
-Once the `scaffolder.ts` router setup file is in place, add the router to
-`packages/backend/src/index.ts`:
-
-```ts
-import scaffolder from './plugins/scaffolder';
-
-const scaffolderEnv = useHotMemoize(module, () => createEnv('scaffolder'));
-
-const apiRouter = Router();
-/* several router .use calls */
-
-/* add this line */
-apiRouter.use('/scaffolder', await scaffolder(scaffolderEnv));
-```
-
-### Adding Templates
-
-At this point the scaffolder backend is installed in your backend package, but
-you will not have any templates available to use. These need to be added to the
-software catalog, as they are represented as entities of kind
-[Template](../software-catalog/descriptor-format.md#kind-template). You can find
-out more about adding templates [here](./adding-templates.md).
-
-To get up and running and try out some templates quickly, you can add some of
-our example templates through static configuration. Add the following to the
-`catalog.locations` section in your `app-config.yaml`:
-
-```yaml
-catalog:
-  locations:
-    # Backstage Example Templates
-    - type: url
-      target: https://github.com/backstage/backstage/blob/master/plugins/scaffolder-backend/sample-templates/react-ssr-template/template.yaml
-    - type: url
-      target: https://github.com/backstage/backstage/blob/master/plugins/scaffolder-backend/sample-templates/springboot-grpc-template/template.yaml
-    - type: url
-      target: https://github.com/backstage/backstage/blob/master/plugins/scaffolder-backend/sample-templates/create-react-app/template.yaml
-    - type: url
-      target: https://github.com/spotify/cookiecutter-golang/blob/master/template.yaml
-```
-
-### Runtime Dependencies / Configuration
-
-For the scaffolder backend plugin to function, you'll need to setup the
-integrations config in your `app-config.yaml`.
-
-You can find help for different providers below.
-
-> Note: Some of this configuration may already be set up as part of your
-> `app-config.yaml`. We're moving away from the duplicated config for
-> authentication in the `scaffolder` section and using `integrations` instead.
-
-#### GitHub
-
-The GitHub access token is retrieved from environment variables via the config.
-The config file needs to specify what environment variable the token is
-retrieved from. Your config should have the following objects.
-
-You can configure who can see the new repositories that the scaffolder creates
-by specifying `visibility` option. Valid options are `public`, `private` and
-`internal`. The `internal` option is for GitHub Enterprise clients, which means
-public within the enterprise.
-
-```yaml
-integrations:
-  github:
-    - host: github.com
-      token: ${GITHUB_TOKEN}
-
-scaffolder:
-  github:
-    visibility: public # or 'internal' or 'private'
-```
-
-#### GitLab
-
-For GitLab, we currently support the configuration of the GitLab publisher and
-allows to configure the private access token and the base URL of a GitLab
-instance:
-
-```yaml
-integrations:
-  gitlab:
-    - host: gitlab.com
-      token: ${GITLAB_TOKEN}
-```
-
-#### Bitbucket
-
-For Bitbucket there are two authentication methods supported. Either `token` or
-a combination of `appPassword` and `username`. It looks like either of the
-following:
-
-```yaml
-integrations:
-  bitbucket:
-    - host: bitbucket.org
-      token: ${BITBUCKET_TOKEN}
-```
-
-or
-
-```yaml
-integrations:
-  bitbucket:
-    - host: bitbucket.org
-      appPassword: ${BITBUCKET_APP_PASSWORD}
-      username: ${BITBUCKET_USERNAME}
-```
-
-#### Azure DevOps
-
-For Azure DevOps we support both the preparer and publisher stage with the
-configuration of a private access token (PAT). For the publisher it's also
-required to define the base URL for the client to connect to the service. This
-will hopefully support on-prem installations as well but that has not been
-verified.
-
-```yaml
-integrations:
-  azure:
-    - host: dev.azure.com
-      token: ${AZURE_TOKEN}
-```
-
-### Running the Backend
-
-Finally, make sure you have a local Docker daemon running, and start up the
-backend with the new configuration:
-
-```bash
-cd packages/backend
-GITHUB_TOKEN=<token> yarn start
-```
-
-If you've also set up the frontend plugin, so you should be ready to go browse
-the templates at [localhost:3000/create](http://localhost:3000/create) now!
-
-### Disabling Docker in Docker situation (Optional)
-
-Software Templates use
-[Cookiecutter](https://github.com/cookiecutter/cookiecutter) as templating
-library. By default it will use the
-[spotify/backstage-cookiecutter](https://github.com/backstage/backstage/blob/37e35b910afc7d1270855aed0ec4718aba366c91/plugins/scaffolder-backend/scripts/Cookiecutter.dockerfile)
-docker image.
-
-If you are running Backstage from a Docker container and you want to avoid
-calling a container inside a container, you can set up Cookiecutter in your own
-image, this will use the local installation instead.
-
-You can do so by including the following lines in the last step of your
-`Dockerfile`:
-
-```Dockerfile
-RUN apt-get update && apt-get install -y python3 python3-pip
-RUN pip3 install cookiecutter
-```
@@ -101,9 +101,7 @@ should have something similar to the below in

 ```ts
 return await createRouter({
-  preparers,
-  templaters,
-  publishers,
+  containerRunner,
  logger,
  config,
  database,
@@ -118,9 +116,7 @@ will set the available actions that the scaffolder has access to.
 ```ts
 const actions = [createNewFileAction()];
 return await createRouter({
-  preparers,
-  templaters,
-  publishers,
+  containerRunner,
  logger,
  config,
  database,
@@ -137,18 +133,17 @@ want to have those as well as your new one, you'll need to do the following:
 import { createBuiltinActions } from '@backstage/plugin-scaffolder-backend';

 const builtInActions = createBuiltinActions({
+  containerRunner,
  integrations,
+  config,
  catalogClient,
-  templaters,
  reader,
 });

 const actions = [...builtInActions, createNewFileAction()];

 return await createRouter({
-  preparers,
-  templaters,
-  publishers,
+  containerRunner,
  logger,
  config,
  database,
@@ -4,8 +4,8 @@ title: Writing Templates
 description: Details around creating your own custom Software Templates
 ---

-Templates are stored in the **Service Catalog** under a kind `Template`. You can
-create your own templates with a small `yaml` definition which describes the
+Templates are stored in the **Software Catalog** under a kind `Template`. You
+can create your own templates with a small `yaml` definition which describes the
 template and it's metadata, along with some input variables that your template
 will need, and then a list of actions which are then executed by the scaffolding
 service.
@@ -62,7 +62,7 @@ spec:
  steps:
    - id: fetch-base
      name: Fetch Base
-      action: fetch:cookiecutter
+      action: fetch:template
      input:
        url: ./template
        values:
@@ -289,8 +289,8 @@ template. These follow the same standard format:
 - id: fetch-base # A unique id for the step
  name: Fetch Base # A title displayed in the frontend
  if: '{{ parameters.name }}' # Optional condition, skip the step if not truthy
-  action: fetch:cookiecutter # an action to call
-  input: # input that is passed as arguments to the action handler
+  action: fetch:template # An action to call
+  input: # Input that is passed as arguments to the action handler
    url: ./template
    values:
      name: '{{ parameters.name }}'
@@ -317,20 +317,20 @@ output:

 ### The templating syntax

-You might have noticed in the examples that there are `{{ }}`, and these are a
-`handlebars` templates for linking and glueing all these different parts of
-`yaml` together. All the form inputs from the `parameters` section, when passed
-to the steps will be available by using the template syntax
-`{{ parameters.something }}`. This is great for passing the values from the form
-into different steps and reusing these input variables. To pass arrays or
-objects use the syntax `{{ json paramaters.something }}` where
-`paramaters.something` is of type `object` or `array` in the `jsonSchema`, such
-as the `nicknames` parameter in the previous example.
+You might have noticed variables wrapped in `{{ }}` in the examples. These are
+`handlebars` template strings for linking and gluing the different parts of the
+template together. All the form inputs from the `parameters` section will be
+available by using this template syntax (for example,
+`{{ parameters.firstName }}` inserts the value of `firstName` from the
+parameters). This is great for passing the values from the form into different
+steps and reusing these input variables. To pass arrays or objects use the
+`json` custom [helper](https://handlebarsjs.com/guide/expressions.html#helpers).
+For example, `{{ json parameters.nicknames }}` will insert the result of calling
+`JSON.stringify` on the value of the `nicknames` parameter.

 As you can see above in the `Outputs` section, `actions` and `steps` can also
-output things. So you can grab that output by using
-`steps.$stepId.output.$property`.
+output things. You can grab that output using `steps.$stepId.output.$property`.

 You can read more about all the `inputs` and `outputs` defined in the actions in
-code part of the `JSONSchema` or you can read more about our built in ones
+code part of the `JSONSchema`, or you can read more about our built in ones
 [here](./builtin-actions.md).
@@ -28,10 +28,10 @@ scratch.
 ### Use the documentation template

 Your working Backstage instance should by default have a documentation template
-added. If not, follow these
-[instructions](../software-templates/installation.md#adding-templates) to add
-the documentation template. The template creates a component with only TechDocs
-configuration and default markdown files as below mentioned in manual
+added. If not, copy the catalog locations from the
+[create-app template](https://github.com/backstage/backstage/blob/master/packages/create-app/templates/default-app/app-config.yaml.hbs)
+to add the documentation template. The template creates a component with only
+TechDocs configuration and default markdown files as below mentioned in manual
 documentation setup, and is otherwise empty.

 ![Documentation Template](../../assets/techdocs/documentation-template.png)
@@ -78,11 +78,6 @@ the repository. The archive does not have any git history attached to it. Also
 it is a compressed file. Hence the file size is significantly smaller than how
 much data git clone has to transfer.

-Caveat: Currently TechDocs sites built using URL Reader will be cached for 30
-minutes which means they will not be re-built if new changes are made within 30
-minutes. This cache invalidation will be replaced by commit timestamp based
-implementation very soon.
-
 ## How to use a custom TechDocs home page?

 ### 1st way: TechDocsCustomHome with a custom configuration
@@ -55,3 +55,20 @@ INFO    -  Start watching changes
 [I 210115 19:00:45 handlers:64] Start detecting changes
 INFO    -  Start detecting changes
 ```
+
+## PlantUML with `svg_object` doesn't render
+
+The [plantuml-markdown](https://pypi.org/project/plantuml-markdown/) MkDocs
+plugin available in
+[`mkdocs-techdocs-core`](https://github.com/backstage/mkdocs-techdocs-core)
+supports different formats for rendering diagrams. TechDocs does however not
+support all of them.
+
+The `svg_object` format renders a diagram as an HTML `<object>` tag but this is
+not allowed as it enables bad actors to inject malicious content into
+documentation pages. See
+[CVE-2021-32661](https://github.com/advisories/GHSA-gg96-f8wr-p89f) for more
+details.
+
+Instead use `svg_inline` which renders as an `<svg>` tag and provides the same
+benefits as `svg_object`.
@@ -370,7 +370,7 @@ techdocs:
    openStackSwift:
      containerName: 'name-of-techdocs-storage-bucket'
      credentials:
-        userName: ${OPENSTACK_SWIFT_STORAGE_USERNAME}
+        username: ${OPENSTACK_SWIFT_STORAGE_USERNAME}
        password: ${OPENSTACK_SWIFT_STORAGE_PASSWORD}
      authUrl: ${OPENSTACK_SWIFT_STORAGE_AUTH_URL}
      keystoneAuthVersion: ${OPENSTACK_SWIFT_STORAGE_AUTH_VERSION}
@@ -56,7 +56,7 @@ For example, adding the theme that we created in the previous section can be
 done like this:

 ```ts
-import { createApp } from '@backstage/core';
+import { createApp } from '@backstage/core-app-api';

 const app = createApp({
  apis: ...,
@@ -94,7 +94,7 @@ here are some useful ones:
 ```python
 yarn start # Start serving the example app, use --check to include type checks and linting

-yarn storybook # Start local storybook, useful for working on components in @backstage/core
+yarn storybook # Start local storybook, useful for working on components in @backstage/core-components

 yarn workspace @backstage/plugin-welcome start # Serve welcome plugin only, also supports --check

@@ -137,7 +137,7 @@ frontend with `yarn start` in one window, and the backend with

 It can often be useful to try out changes to the packages in the main Backstage
 repo within your own app. For example if you want to make modifications to
-`@backstage/core` and try them out in your app.
+`@backstage/core-plugin-api` and try them out in your app.

 To link in external packages, add them to your `package.json` and `lerna.json`
 workspace paths. These can be either relative or absolute paths with or without
@@ -147,7 +147,7 @@ globs. For example:
 "packages": [
  "packages/*",
  "plugins/*",
-  "../backstage/packages/core", // New path added to work on @backstage/core
+  "../backstage/packages/core-plugin-api", // New path added to work on @backstage/core-plugin-api
 ],
 ```

@@ -157,9 +157,10 @@ Then reinstall packages to make yarn set up symlinks:
 yarn install
 ```

-With this in place you can now modify the `@backstage/core` package within the
-main repo, and have those changes be reflected and tested in your app. Simply
-run your app using `yarn dev` (or `yarn start` for just frontend) as normal.
+With this in place you can now modify the `@backstage/core-plugin-api` package
+within the main repo, and have those changes be reflected and tested in your
+app. Simply run your app using `yarn dev` (or `yarn start` for just frontend) as
+normal.

 Note that for backend packages you need to make sure that linked packages are
 not dependencies of any non-linked package. If you for example want to work on
@@ -127,18 +127,17 @@ are separated out into their own folder, see further down.
  used by the backend, we chose to separate `config` and `config-loader` into
  two different packages.

- [`core/`](https://github.com/backstage/backstage/tree/master/packages/core) -
+- [`core-app-api/`](https://github.com/backstage/backstage/tree/master/packages/core-app-api) -
+  This package contains the core APIs that are used to wire together Backstage
+  apps.
+
+- [`core-components/`](https://github.com/backstage/backstage/tree/master/packages/core-components) -
  This package contains our visual React components, some of which you can find
  in
  [plugin examples](https://backstage.io/storybook/?path=/story/plugins-examples--plugin-with-data).
-  Apart from that it re-exports everything from [`core-api`] so that users only
-  need to rely on one package.

- [`core-api/`](https://github.com/backstage/backstage/tree/master/packages/core-api) -
-  This package contains APIs and definitions of such. It is it's own package
-  because we needed to split our `test-utils` package. It's an implementation
-  detail that we try to hide from our users, and no one should have to depend on
-  it directly.
+- [`core-plugin-api/`](https://github.com/backstage/backstage/tree/master/packages/core-plugin-api) -
+  This package contains the core APIs that are used to build Backstage plugins.

 - [`create-app/`](https://github.com/backstage/backstage/tree/master/packages/create-app) -
  An CLI to specifically scaffold a new Backstage App. It does so by using a
@@ -6,8 +6,8 @@ sidebar_label: Locations
 description: Integrating source code stored in Azure DevOps into the Backstage catalog
 ---

-The Azure integration supports loading catalog entities from Azure DevOps.
-Entities can be added to
+The Azure DevOps integration supports loading catalog entities from Azure
+DevOps. Entities can be added to
 [static catalog configuration](../../features/software-catalog/configuration.md),
 or registered with the
 [catalog-import](https://github.com/backstage/backstage/tree/master/plugins/catalog-import)
@@ -0,0 +1,14 @@
+---
+id: org
+title: Microsoft Azure Active Directory Organizational Data
+sidebar_label: Org Data
+# prettier-ignore
+description: Importing users and groups from a Microsoft Azure Active Directory into Backstage
+---
+
+The Backstage catalog can be set up to ingest organizational data - users and
+teams - directly from an tenant in Microsoft Azure Active Directory via the
+Microsoft Graph API.
+
+More details on this are available in the
+[README of the `@backstage/plugin-catalog-backend-module-msgraph` package](https://github.com/backstage/backstage/blob/master/plugins/catalog-backend-module-msgraph/README.md).
@@ -14,19 +14,25 @@ entities that mirror your org setup.

 ## Installation

-The processor that performs the import, `LdapOrgReaderProcessor`, comes
-installed with the default setup of Backstage.
+1. The processor is not installed by default, therefore you have to add a
+   dependency to `@backstage/plugin-catalog-backend-module-ldap` to your backend
+   package.

-If you replace the set of processors in your installation using that facility of
-the catalog builder class, you can import and add it as follows.
+```bash
+# From your Backstage root directory
+cd packages/backend
+yarn add @backstage/plugin-catalog-backend-module-ldap
+```

-```ts
-// Typically in packages/backend/src/plugins/catalog.ts
-import { LdapOrgReaderProcessor } from '@backstage/plugin-catalog-backend';
+2. The `LdapOrgReaderProcessor` is not registered by default, so you have to
+   register it in the catalog plugin:

-builder.replaceProcessors(
-  LdapOrgReaderProcessor.fromConfig(config, { logger }),
-  // ...
+```typescript
+// packages/backend/src/plugins/catalog.ts
+builder.addProcessor(
+  LdapOrgReaderProcessor.fromConfig(config, {
+    logger,
+  }),
 );
 ```

@@ -116,8 +122,8 @@ The DN under which users are stored, e.g.
 #### users.options

 The search options to use when sending the query to the server, when reading all
-users. All of the options are shown below, with their default values, but they
-are all optional.
+users. All the options are shown below, with their default values, but they are
+all optional.

 ```yaml
 options:
@@ -152,8 +158,8 @@ set:

 Mappings from well known entity fields, to LDAP attribute names. This is where
 you are able to define how to interpret the attributes of each LDAP result item,
-and to move them into the corresponding entity fields. All of the options are
-shown below, with their default values, but they are all optional.
+and to move them into the corresponding entity fields. All the options are shown
+below, with their default values, but they are all optional.

 If you leave out an optional mapping, it will still be copied using that default
 value. For example, even if you do not put in the field `displayName` in your
@@ -198,8 +204,8 @@ The DN under which groups are stored, e.g.
 #### groups.options

 The search options to use when sending the query to the server, when reading all
-groups. All of the options are shown below, with their default values, but they
-are all optional.
+groups. All the options are shown below, with their default values, but they are
+all optional.

 ```yaml
 options:
@@ -276,3 +282,35 @@ map:
  # the spec.children field of the entity.
  members: member
 ```
+
+## Customize the Processor
+
+In case you want to customize the ingested entities, the
+`LdapOrgReaderProcessor` allows to pass transformers for users and groups.
+
+1. Create a transformer:
+
+```ts
+export async function myGroupTransformer(
+  vendor: LdapVendor,
+  config: GroupConfig,
+  group: SearchEntry,
+): Promise<GroupEntity | undefined> {
+  // Transformations may change namespace, change entity naming pattern, fill
+  // profile with more or other details...
+
+  // Create the group entity on your own, or wrap the default transformer
+  return await defaultGroupTransformer(vendor, config, group);
+}
+```
+
+2. Configure the processor with the transformer:
+
+```ts
+builder.addProcessor(
+  LdapOrgReaderProcessor.fromConfig(config, {
+    logger,
+    groupTransformer: myGroupTransformer,
+  }),
+);
+```
@@ -25,7 +25,7 @@ different ways.

 The following diagram shows how Backstage might look when deployed inside a
 company which uses the Tech Radar plugin, the Lighthouse plugin, the CircleCI
-plugin and the service catalog.
+plugin and the software catalog.

 There are 3 main components in this architecture:

@@ -142,7 +142,7 @@ Its architecture looks like this:

 ![lighthouse plugin backed to microservice and database](../assets/architecture-overview/lighthouse-plugin-architecture.png)

-The service catalog in Backstage is another example of a service backed plugin.
+The software catalog in Backstage is another example of a service backed plugin.
 It retrieves a list of services, or "entities", from the Backstage Backend
 service and renders them in a table for the user.

@@ -22,8 +22,8 @@ Our idea was to centralize and simplify end-to-end software development with an
 abstraction layer that sits on top of all of our infrastructure and developer
 tooling. That’s Backstage.

-It’s a developer portal powered by a centralized service catalog — with a plugin
-architecture that makes it endlessly extensible and customizable.
+It’s a developer portal powered by a centralized software catalog — with a
+plugin architecture that makes it endlessly extensible and customizable.

 Manage all your services, software, tooling, and testing in Backstage. Start
 building a new microservice using an automated template in Backstage. Create,
@@ -11,7 +11,7 @@ terminology below for clarity and consistency when discussing Backstage.

 ### Authentication Glossary

-This [page](./auth/glossary.md) directs to the terms and phrases related to
+This [page](../auth/glossary.md) directs to the terms and phrases related to
 authentication and identity section of Backstage.

 ### Backstage User Profiles
@@ -22,7 +22,7 @@ We have divided the project into three high-level _phases_:
  [UX patterns and components](https://backstage.io/storybook) help ensure a
  consistent experience between tools.

- 🐢 **Phase 2:** Service Catalog
+- 🐢 **Phase 2:** Software Catalog
  ([alpha released](https://backstage.io/blog/2020/06/22/backstage-service-catalog-alpha)) -
  With a single catalog, Backstage makes it easy for a team to manage ten
  services — and makes it possible for your company to manage thousands of them.
@@ -120,13 +120,13 @@ Chances are that someone will jump in and help build it.
 - [TechDocs v1](https://backstage.io/blog/2020/09/08/announcing-tech-docs)
 - [Plugin marketplace](https://backstage.io/plugins)
 - [Improved and move documentation to backstage.io](https://backstage.io/docs/overview/what-is-backstage)
- [Backstage Service Catalog (alpha)](https://backstage.io/blog/2020/06/22/backstage-service-catalog-alpha)
+- [Backstage Software Catalog (alpha)](https://backstage.io/blog/2020/06/22/backstage-service-catalog-alpha)
 - [Backstage Software Templates (alpha)](https://backstage.io/blog/2020/08/05/announcing-backstage-software-templates)
 - [Make it possible to add custom auth providers](https://backstage.io/blog/2020/07/01/how-to-enable-authentication-in-backstage-using-passport)
 - [TechDocs v0](https://github.com/backstage/backstage/milestone/15)
 - CI plugins: CircleCI, Jenkins, GitHub Actions and TravisCI
 - [Service API documentation](https://github.com/backstage/backstage/pull/1737)
- Backstage Service Catalog can read from: GitHub, GitLab,
+- Backstage Software Catalog can read from: GitHub, GitLab,
  [Bitbucket](https://github.com/backstage/backstage/pull/1938)
 - Support auth providers: Google, Okta, GitHub, GitLab,
  [auth0](https://github.com/backstage/backstage/pull/1611),
@@ -106,16 +106,6 @@ Used to load in static configuration, mainly for use by the CLI and

 Stability: `1`. Mainly intended for internal use.

-### `core` [GitHub](https://github.com/backstage/backstage/tree/master/packages/core/)
-
-The `@backstage/core` and `@backstage/core-api` packages are being phased out
-and replaced by other `@backstage/core-*` packages. They are still in use but
-will not receive any breaking changes.
-
-### `core-api` [GitHub](https://github.com/backstage/backstage/tree/master/packages/core-api/)
-
-Stability: See `@backstage/core` above
-
 ### `core-app-api` [GitHub](https://github.com/backstage/backstage/tree/master/packages/core-app-api/)

 The APIs used exclusively in the app, such as `createApp` and the system icons.
@@ -194,8 +184,9 @@ Stability: `2`
 ### `test-utils-core` [GitHub](https://github.com/backstage/backstage/tree/master/packages/test-utils-core/)

 Internal testing utilities that are separated out for usage in
-@backstage/core-api. All exports are re-exported by @backstage/test-utils. This
-package should not be depended on directly.
+@backstage/core-app-api and @backstage/core-plugin-api. All exports are
+re-exported by @backstage/test-utils. This package should not be depended on
+directly.

 Stability: See @backstage/test-utils

@@ -218,9 +209,6 @@ Stability: `1`

 ## Plugins

-Plugins are rarely marked as stable as the `@backstage/core` plugin API is under
-heavy development.
-
 Many backend plugins are split into "REST API" and "TypeScript Interface"
 sections. The "TypeScript Interface" refers to the API used to integrate the
 plugin into the backend.
@@ -344,8 +332,7 @@ Stability: `1`
 The backend scaffolder plugin that provides an implementation for templates in
 the catalog.

-Stability: `1`. There is planned work to rework the scaffolder in
-https://github.com/backstage/backstage/issues/2771.
+Stability: `2`.

 ### `tech-radar` [GitHub](https://github.com/backstage/backstage/tree/master/plugins/tech-radar/)

@@ -2,13 +2,13 @@
 id: what-is-backstage
 title: What is Backstage?
 # prettier-ignore
-description: Backstage is an open platform for building developer portals. Powered by a centralized service catalog, Backstage restores order to your microservices and infrastructure
+description: Backstage is an open platform for building developer portals. Powered by a centralized software catalog, Backstage restores order to your microservices and infrastructure
 ---

-![service-catalog](https://backstage.io/blog/assets/6/header.png)
+![software-catalog](https://backstage.io/blog/assets/6/header.png)

 [Backstage](https://backstage.io/) is an open platform for building developer
-portals. Powered by a centralized service catalog, Backstage restores order to
+portals. Powered by a centralized software catalog, Backstage restores order to
 your microservices and infrastructure and enables your product teams to ship
 high-quality code quickly — without compromising autonomy.

@@ -17,7 +17,7 @@ to create a streamlined development environment from end to end.

 Out of the box, Backstage includes:

- [Backstage Service Catalog](../features/software-catalog/index.md) for
+- [Backstage Software Catalog](../features/software-catalog/index.md) for
  managing all your software (microservices, libraries, data pipelines,
  websites, ML models, etc.)

@@ -99,7 +99,7 @@ import carmen from './plugins/carmen';
 async function main() {
  // ...
  const carmenEnv = useHotMemoize(module, () => createEnv('carmen'));
-  apiRouter.use('/carmen', await carmen(badgesEnv));
+  apiRouter.use('/carmen', await carmen(carmenEnv));
 ```

 After you start the backend (e.g. using `yarn start-backend` from the repo
@@ -111,3 +111,42 @@ curl localhost:7000/api/carmen/health
 ```

 This should return `{"status":"ok"}` like before. Success!
+
+## Making Use of a Database
+
+The Backstage backend comes with a builtin facility for SQL database access.
+Most plugins that have persistence needs will choose to make use of this
+facility, so that Backstage operators can manage database needs uniformly.
+
+As part of the environment object that is passed to your `createPlugin`
+function, there is a `database` field. You can use that to get a
+[Knex](http://knexjs.org/) connection object.
+
+```ts
+// in packages/backend/src/plugins/carmen.ts
+export default async function createPlugin(env: PluginEnvironment) {
+  const db: Knex<any, unknown[]> = await env.database.getClient();
+
+  // You will then pass this client into your actual plugin implementation
+  // code, maybe similar to the following:
+  const model = new CarmenDatabaseModel(db);
+  return await createRouter({
+    model: model,
+    logger: env.logger,
+  });
+}
+```
+
+You may note that the `getClient` call has no parameters. This is because all
+plugin database needs are configured under the `backend.database` config key of
+your `app-config.yaml`. The framework may even make sure behind the scenes that
+the logical database is created automatically if it doesn't exist, based on
+rules that the Backstage operator decides on.
+
+The framework does not handle database schema migrations for you, however. The
+builtin plugins in the main repo have chosen to use the Knex library to manage
+schema migrations as well, but you can do so in any manner that you see fit.
+
+See the [Knex library documentation](http://knexjs.org/) for examples and
+details on how to write schema migrations and perform SQL queries against your
+database..
@@ -511,10 +511,11 @@ clarify intent. Refer to the following table to formulate the new name:
 ## Porting Existing Apps

 The first step of porting any app is to replace the root `Routes` component with
-`FlatRoutes` from `@backstage/core`. As opposed to the `Routes` component,
-`FlatRoutes` only considers the first level of `Route` components in its
-children, and provides any additional children to the outlet of the route. It
-also removes the need to append `"/*"` to paths, as it is added automatically.
+`FlatRoutes` from `@backstage/core-app-api`. As opposed to the `Routes`
+component, `FlatRoutes` only considers the first level of `Route` components in
+its children, and provides any additional children to the outlet of the route.
+It also removes the need to append `"/*"` to paths, as it is added
+automatically.

 ```diff
 const AppRoutes = () => (
@@ -22,9 +22,9 @@ yarn create-plugin
 This will create a new Backstage Plugin based on the ID that was provided. It
 will be built and added to the Backstage App automatically.

-> If `yarn start` is already running you should be able to see the default page
-> for your new plugin directly by navigating to
-> `http://localhost:3000/my-plugin`.
+> If the Backstage App is already running (with `yarn start` or `yarn dev`) you
+> should be able to see the default page for your new plugin directly by
+> navigating to `http://localhost:3000/my-plugin`.

 ![](../assets/my-plugin_screenshot.png)

@@ -32,7 +32,7 @@ You can also serve the plugin in isolation by running `yarn start` in the plugin
 directory. Or by using the yarn workspace command, for example:

 ```bash
-yarn workspace @backstage/plugin-welcome start # Also supports --check
+yarn workspace @backstage/my-plugin start # Also supports --check
 ```

 This method of serving the plugin provides quicker iteration speed and a faster
@@ -43,6 +43,10 @@ root of the project which you can then use as an `include` in your
 `app-config.yaml`. You can go ahead and
 [skip ahead](#including-in-integrations-config) if you've already got an app.

+Note that the created app will have a webhook that is disabled by default and
+points to `smee.io`, which is intended for local development. There's also
+currently no part of Backstage that makes use of the webhook.
+
 ### GitHub Enterprise

 You have to create the GitHub Application manually using these
@@ -104,3 +108,12 @@ privateKey: |

 This will result in backstage preventing the use of any installation that is not
 within the allow list.
+
+### Permissions for pull requests
+
+These are the minimum permissions required for creating a pull request with
+Backstage software templates:
+
+- Read and Write permissions for `Contents`.
+- Read and write permissions for `Pull Requests` and `Issues`.
+- Read permissions on `Metadata`.
@@ -28,9 +28,9 @@ This helps the community know what plugins are in development.
 You can also use this process if you have an idea for a good plugin but you hope
 that someone else will pick up the work.

-## Integrate into the Service Catalog
+## Integrate into the Software Catalog

 If your plugin isn't supposed to live as a standalone page, but rather needs to
-be presented as a part of a Service Catalog (e.g. a separate tab or a card on an
-"Overview" tab), then check out
-[the instruction](integrating-plugin-into-service-catalog.md) on how to do it.
+be presented as a part of a Software Catalog (e.g. a separate tab or a card on
+an "Overview" tab), then check out
+[the instruction](integrating-plugin-into-software-catalog.md) on how to do it.
@@ -1,7 +1,7 @@
 ---
-id: integrating-plugin-into-service-catalog
-title: Integrate into the Service Catalog
-description: How to integrate a plugin into service catalog
+id: integrating-plugin-into-software-catalog
+title: Integrate into the Software Catalog
+description: How to integrate a plugin into software catalog
 ---

 > This is an advanced use case and currently is an experimental feature. Expect
@@ -36,7 +36,7 @@ to avoid import cycles, for example like this:

 ```tsx
 /* src/routes.ts */
-import { createRouteRef } from '@backstage/core';
+import { createRouteRef } from '@backstage/core-plugin-api';

 // Note: This route ref is for internal use only, don't export it from the plugin
 export const rootRouteRef = createRouteRef({
@@ -46,11 +46,11 @@ export const rootRouteRef = createRouteRef({

 Now that we have a `RouteRef`, we import it into `src/plugin.ts`, create our
 plugin instance with `createPlugin`, as well as create and wrap our routable
-extension using `createRoutableExtension` from `@backstage/core`:
+extension using `createRoutableExtension` from `@backstage/core-plugin-api`:

 ```tsx
 /* src/plugin.ts */
-import { createPlugin, createRouteRef } from '@backstage/core';
+import { createPlugin, createRouteRef } from '@backstage/core-plugin-api';
 import ExampleComponent from './components/ExampleComponent';

 // Create a plugin instance and export this from your plugin package
@@ -57,7 +57,10 @@ package.json to declare the plugin dependencies, metadata and scripts.
 In the `src` folder we get to the interesting bits. Check out the `plugin.ts`:

 ```jsx
-import { createPlugin, createRoutableExtension } from '@backstage/core';
+import {
+  createPlugin,
+  createRoutableExtension,
+} from '@backstage/core-plugin-api';

 import { rootRouteRef } from './routes';

@@ -0,0 +1,274 @@
+---
+id: url-reader
+title: URL Reader
+sidebar_label: URL Reader
+# prettier-ignore
+description: URL Reader is a backend core API responsible for reading files from external locations.
+---
+
+## Concept
+
+Some of the core plugins of Backstage have to read files from an external
+location. [Software Catalog](../features/software-catalog/index.md) has to read
+the [`catalog-info.yaml`](../features/software-catalog/descriptor-format.md)
+entity descriptor files to register and track an entity.
+[Software Templates](../features/software-templates/index.md) have to download
+the template skeleton files before creating a new component.
+[TechDocs](../features/techdocs/README.md) has to download the markdown source
+files before generating a documentation site.
+
+Since, the requirement for reading files is so essential for Backstage plugins,
+the
+[`@backstage/backend-common`](https://github.com/backstage/backstage/tree/master/packages/backend-common)
+package provides a dedicated API for reading from such URL based remote
+locations like GitHub, GitLab, Bitbucket, Google Cloud Storage, etc. This is
+commonly referred to as "URL Reader". It takes care of making authenticated
+requests to the remote host so that private files can be read securely. If users
+have [GitHub App based authentication](github-apps.md) set up, URL Reader even
+refreshes the token, to avoid reaching the GitHub API rate limit.
+
+As a result, plugin authors do not have to worry about any of these problems
+when trying to read files.
+
+## Interface
+
+When the Backstage backend starts, a new instance of URL Reader is created. You
+can see this in the index file of your Backstage backend
+i.e.`packages/backend/src/index.ts`.
+[Example](https://github.com/backstage/backstage/blob/ebbe91dbe79038a61d35cf6ed2d96e0e0d5a15f3/packages/backend/src/index.ts#L57)
+
+```ts
+// File: packages/backend/src/index.ts
+
+import { URLReaders } from '@backstage/backend-common';
+
+function makeCreateEnv(config: Config) {
+  // ....
+  const reader = UrlReaders.default({ logger, config });
+  //
+}
+```
+
+This instance contains all
+[the default URL Reader providers](https://github.com/backstage/backstage/blob/master/packages/backend-common/src/reading/UrlReaders.ts)
+in the backend-common package including GitHub, GitLab, Bitbucket, Azure, Google
+GCS. As the need arises, more URL Readers are being written to support different
+providers.
+
+The generic interface of a URL Reader instance looks like this.
+
+```ts
+export type UrlReader = {
+  /* Used to read a single file and return its content. */
+  read(url: string): Promise<Buffer>;
+  /**
+   * A replacement for the read method that supports options and complex responses.
+   *
+   * Use this whenever it is available, as the read method will be deprecated and
+   * eventually removed in the future.
+   */
+  readUrl?(url: string, options?: ReadUrlOptions): Promise<ReadUrlResponse>;
+
+  /* Used to read a file tree and download as a directory. */
+  readTree(url: string, options?: ReadTreeOptions): Promise<ReadTreeResponse>;
+  /* Used to search a file in a tree. */
+  search(url: string, options?: SearchOptions): Promise<SearchResponse>;
+};
+```
+
+## Using a URL Reader inside a plugin
+
+The `reader` instance is available in the backend plugin environment and passed
+on to all the backend plugins. You can see an
+[example](https://github.com/backstage/backstage/blob/b0be185369ebaad22255b7cdf18535d1d4ffd0e7/packages/backend/src/plugins/techdocs.ts#L31).
+When any of the methods on this instance is called with a URL, URL Reader
+extracts the host for that URL (e.g. `github.com`, `ghe.mycompany.com`, etc.).
+Using the
+[`@backstage/integration`](https://github.com/backstage/backstage/tree/master/packages/integration)
+package, it looks inside the
+[`integrations:`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/app-config.yaml#L134-L158)
+config of the `app-config.yaml` to find out how to work with the host based on
+the configs provided like authentication token, API base URL, etc.
+
+Make sure your plugin-specific backend file at
+`packages/backend/src/plugins/<PLUGIN>.ts` is forwarding the `reader` instance
+passed on as the `PluginEnvironment` to the actual plugin's `createRouter`
+function. See how this is done in
+[Catalog](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend/src/plugins/catalog.ts#L25-L27)
+and
+[TechDocs](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend/src/plugins/techdocs.ts#L31-L36)
+backend plugins.
+
+Once the reader instance is available inside the plugin, one of its methods can
+directly be used with a URL. Some example usages -
+
+- [`read`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/plugins/catalog-backend/src/ingestion/processors/codeowners/read.ts#L24-L33) -
+  Catalog using the `read` method to read the CODEOWNERS file in a repository.
+- [`readTree`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/techdocs-common/src/helpers.ts#L198-L220) -
+  TechDocs using the `readTree` method to download markdown files in order to
+  generate the documentation site.
+- [`readTree`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/techdocs-common/src/stages/prepare/url.ts#L33-L54) -
+  TechDocs using `NotModifiedError` to maintain cache and speed up and limit the
+  number of requests.
+- [`search`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/plugins/catalog-backend/src/ingestion/processors/UrlReaderProcessor.ts#L88-L108) -
+  Catalog using the `search` method to find files for a location URL containing
+  a glob pattern.
+
+## Writing a new URL Reader
+
+If the available URL Readers are not sufficient for your use case and you want
+to add a new URL Reader for any other provider, you are most welcome to
+contribute one!
+
+Feel free to use the
+[GitHub URL Reader](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/GithubUrlReader.ts)
+as a source of inspiration.
+
+### 1. Add an integration
+
+The provider for your new URL Reader can also be called an "integration" in
+Backstage. The `integrations:` section of your Backstage `app-config.yaml`
+config file is supposed to be the place where a Backstage integrator defines the
+host URL for the integration, authentication details and other integration
+related configurations.
+
+The `@backstage/integration` package is where most of the integration specific
+code lives, so that it is shareable across Backstage. Functions like "read the
+integrations config and process it", "construct headers for authenticated
+requests to the host" or "convert a plain file URL into its API URL for
+downloading the file" would live in this package.
+
+### 2. Create the URL Reader
+
+Create a new class which implements the
+[`UrlReader` type](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/types.ts#L21-L28)
+inside `@backstage/backend-common`. Create and export a static `factory` method
+which reads the integration config and returns a map of host URLs the new reader
+should be used for. See the
+[GitHub URL Reader](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/GithubUrlReader.ts#L50-L63)
+for example.
+
+### 3. Implement the methods
+
+We want to make sure all URL Readers behave in the same way. Hence if possible,
+all the methods of the `UrlReader` interface should be implemented. However it
+is okay to start by implementing just one of them and create issues for the
+remaining.
+
+#### read
+
+NOTE: Use `readUrl` instead of `read`.
+
+`read` method expects a user-friendly URL, something which can be copied from
+the browser naturally when a person is browsing the provider in their browser.
+
+- ✅ Valid URL :
+  `https://github.com/backstage/backstage/blob/master/ADOPTERS.md`
+- ❌ Not a valid URL :
+  `https://raw.githubusercontent.com/backstage/backstage/master/ADOPTERS.md`
+- ❌ Not a valid URL : `https://github.com/backstage/backstage/ADOPTERS.md`
+
+Upon receiving the URL, `read` converts the user-friendly URL into an API URL
+which can be used to request the provider's API.
+
+`read` then makes an authenticated request to the provider API and returns the
+file's content.
+
+#### readUrl
+
+`readUrl` is a new interface that allows complex response objects and is
+intended to replace the `read` method. This new method is currently optional to
+implement which allows for a soft migration to `readUrl` instead of `read` in
+the future.
+
+#### readTree
+
+`readTree` method also expects user-friendly URLs similar to `read` but the URL
+should point to a tree (could be the root of a repository or even a
+sub-directory).
+
+- ✅ Valid URL : `https://github.com/backstage/backstage`
+- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master`
+- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/docs`
+
+Using the provider's API documentation, find out an API endpoint which can be
+used to download either a zip or a tarball. You can download the entire tree
+(e.g. a repository) and filter out in case the user is expecting only a
+sub-tree. But some APIs are smart enough to accept a path and return only a
+sub-tree in the downloaded archive.
+
+#### search
+
+`search` method expects a glob pattern of a URL and returns a list of files
+matching the query.
+
+- ✅ Valid URL :
+  `https://github.com/backstage/backstage/blob/master/**/catalog-info.yaml`
+- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/**/*.md`
+- ✅ Valid URL :
+  `https://github.com/backstage/backstage/blob/master/*/package.json`
+- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/READM`
+
+The core logic of `readTree` can be used here to extract all the files inside
+the tree and return the files matching the pattern in the `url`.
+
+### 4. Add to available URL Readers
+
+There are two ways to make your new URL Reader available for use.
+
+You can choose to make it open source, by updating the
+[`default` factory](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/UrlReaders.ts#L62-L81)
+method of URL Readers.
+
+But for something internal which you don't want to make open source, you can
+update your `packages/backend/src/index.ts` file and update how the `reader`
+instance is created.
+
+```ts
+// File: packages/backend/src/index.ts
+const reader = UrlReaders.default({
+  logger: root,
+  config,
+  // This is where your internal URL Readers would go.
+  factories: [myCustomReader.factory],
+});
+```
+
+### 5. Caching
+
+All of the methods above support an ETag based caching. If the method is called
+without an `etag`, the response contains an ETag of the resource (should ideally
+forward the ETag returned by the provider). If the method is called with an
+`etag`, it first compares the ETag and returns a `NotModifiedError` in case the
+resource has not been modified. This approach is very similar to the actual
+[ETag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag) and
+[If-None-Match](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match)
+HTTP headers.
+
+### 6. Debugging
+
+When debugging one of the URL Readers, you can straightforward use the
+[`reader` instance created](https://github.com/backstage/backstage/blob/ebbe91dbe79038a61d35cf6ed2d96e0e0d5a15f3/packages/backend/src/index.ts#L57)
+when the backend starts and call one of the methods with your debugging URL.
+
+```ts
+// File: packages/backend/src/index.ts
+
+async function main() {
+  // ...
+  const createEnv = makeCreateEnv(config);
+
+  const testReader = createEnv('test-url-reader').reader;
+  const response = await testReader.readUrl(
+    'https://github.com/backstage/backstage/blob/master/catalog-info.yaml',
+  );
+  console.log((await response.buffer()).toString());
+  // ...
+}
+```
+
+This will be run every time you restart the backend. Note that after any change
+in the URL Reader code, you need to kill the backend and restart, since the
+`reader` instance is memoized and does not update on hot module reloading. Also,
+there are a lot of unit tests written for the URL Readers, which you can make
+use of.
@@ -1,5 +1,5 @@
 /*
- * Copyright 2020 Spotify AB
+ * Copyright 2020 The Backstage Authors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
@@ -11,7 +11,7 @@ can use this to split out logic in your code for manual A/B testing, etc.
 Here's a code sample:

 ```typescript
-import { createPlugin } from '@backstage/core';
+import { createPlugin } from '@backstage/core-plugin-api';

 export default createPlugin({
  id: 'plugin-name',
@@ -29,7 +29,7 @@ To inspect the state of a feature flag inside your plugin, you can use the
 ```tsx
 import React from 'react';
 import { Button } from '@material-ui/core';
-import { featureFlagsApiRef, useApi } from '@backstage/core';
+import { featureFlagsApiRef, useApi } from '@backstage/core-plugin-api';

 const ExamplePage = () => {
  const featureFlags = useApi(featureFlagsApiRef);
@@ -30,7 +30,7 @@ type PluginHooks = {
 Showcasing adding a feature flag.

 ```jsx
-import { createPlugin } from '@backstage/core';
+import { createPlugin } from '@backstage/core-plugin-api';

 export default createPlugin({
  id: 'new-plugin',
@@ -0,0 +1,204 @@
+---
+id: configuring-plugin-databases
+title: Configuring Plugin Databases
+# prettier-ignore
+description: Guide on how to configure Backstage databases.
+---
+
+This guide covers a variety of production persistence use cases which are
+supported out of the box by Backstage. The database manager allows the developer
+to set the client and database connection details on a per plugin basis in
+addition to the base client and connection configuration. This means that you
+can use a SQLite 3 in-memory database for a specific plugin whilst using
+PostgreSQL for everything else and so on.
+
+By default, Backstage uses automatically created databases for each plugin whose
+names follow the `backstage_plugin_<pluginId>` pattern, e.g.
+`backstage_plugin_auth`. You can configure a different database name prefix for
+use cases where you have multiple deployments running on a shared database
+instance or cluster.
+
+With infrastructure defined as code or data (Terraform, AWS CloudFormation,
+etc.), you may have database credentials which lack permissions to create new
+databases or you do not have control over the database names. In these
+instances, you can set the database name and connection information on a per
+plugin basis as mentioned earlier.
+
+Backstage supports all of these use cases with the `DatabaseManager` provided by
+`@backstage/backend-common`. We will now cover how to use and configure
+Backstage's databases.
+
+## Prerequisites
+
+### Dependencies
+
+Please ensure the appropriate database drivers are installed in your `backend`
+package. If you intend to use both `postgres` and `sqlite3`, you can install
+both of them.
+
+```sh
+cd packages/backend
+
+# install pg if you need postgres
+yarn add pg
+
+# install sqlite3 if you intend to set it as the client
+yarn add sqlite3
+```
+
+From an operational perspective, you only need to install drivers for clients
+that are actively used.
+
+### Database Manager
+
+Existing Backstage instances should be updated to use `DatabaseManager` from
+`@backstage/backend-common` in your `packages/backend/src/index.ts` file, the
+`SingleConnectionDatabaseManager` has been deprecated. Import the manager and
+update the references as shown below if this is not the case:
+
+```diff
+import {
+-  SingleConnectionDatabaseManager,
+  DatabaseManager,
+} from '@backstage/backend-common';
+
+// ...
+
+function makeCreateEnv(config: Config) {
+  // ...
+-  const databaseManager = SingleConnectionDatabaseManager.fromConfig(config);
+  const databaseManager = DatabaseManager.fromConfig(config);
+  // ...
+}
+```
+
+## Configuration
+
+You should set the base database client and connection information in your
+`app-config.yaml` (or equivalent) file. The base client and configuration is
+used as the default which is extended for each plugin with the same or unset
+client type. If a client type is specified for a specific plugin which does not
+match the base client, the configuration set for the plugin will be used as is
+without extending the base configuration.
+
+Client type and configuration for plugins need to be defined under
+**`backend.database.plugin.<pluginId>`**. As an example, `catalog` is the
+`pluginId` for the catalog plugin and any configuration defined under that block
+is specific to that plugin. We will now explore more detailed example
+configurations below.
+
+### Minimal In-Memory Configuration
+
+In the example below, we are using `sqlite3` in-memory databases for all
+plugins. You may want to use this configuration for testing or other non-durable
+use cases.
+
+```yaml
+backend:
+  database:
+    client: sqlite3
+    connection: ':memory:'
+```
+
+### PostgreSQL
+
+The example below uses PostgreSQL (`pg`) as the database client for all plugins.
+The `auth` plugin uses a user defined database name instead of the automatically
+generated one which would have been `backstage_plugin_auth`.
+
+```yaml
+backend:
+  database:
+    client: pg
+    connection:
+      host: some.example-pg-instance.tld
+      user: postgres
+      password: password
+      port: 5432
+    plugin:
+      auth:
+        connection:
+          database: pg_auth_set_by_user
+```
+
+### Custom Database Name Prefix
+
+The configuration below uses `example_prefix_` as the database name prefix
+instead of `backstage_plugin_`. Plugins such as `auth` and `catalog` will use
+databases named `example_prefix_auth` and `example_prefix_catalog` respectively.
+
+```yaml
+backend:
+  database:
+    client: pg
+    connection:
+      host: some.example-pg-instance.tld
+      user: postgres
+      password: password
+      port: 5432
+    prefix: 'example_prefix_'
+```
+
+### Connection Configuration Per Plugin
+
+Both `auth` and `catalog` use connection configuration with different
+credentials and database names. This type of configuration can be useful for
+environments with infrastructure as code or data which may provide randomly
+generated credentials and/or database names.
+
+```yaml
+backend:
+  database:
+    client: pg
+    connection: 'postgresql://some.example-pg-instance.tld:5432'
+    plugin:
+      auth:
+        connection: 'postgresql://fort:knox@some.example-pg-instance.tld:5432/unwitting_fox_jumps'
+      catalog:
+        connection: 'postgresql://bank:reserve@some.example-pg-instance.tld:5432/shuffle_ransack_playback'
+```
+
+### PostgreSQL and SQLite 3
+
+The example below uses PostgreSQL (`pg`) as the database client for all plugins
+except the `auth` plugin which uses `sqlite3`. As the `auth` plugin's client
+type is different from the base client type, the connection configuration for
+`auth` is used verbatim without extending the base configuration for PostgreSQL.
+
+```yaml
+backend:
+  database:
+    client: pg
+    connection: 'postgresql://foo:bar@some.example-pg-instance.tld:5432'
+    plugin:
+      auth:
+        client: sqlite3
+        connection: ':memory:'
+```
+
+## Check Your Databases
+
+The `DatabaseManager` will attempt to create the databases if they do not exist.
+If you have set credentials per plugin because the credentials in the base
+configuration do not have permissions to create databases, you must ensure they
+exist before starting the service. The service will not be able to create them,
+it can only use them.
+
+### Privileges
+
+As Backstage attempts to check if the database exists, you may need to grant
+privileges to list or show databases for a given user. For PostgreSQL, you would
+grant the following:
+
+```postgres
+GRANT SELECT ON pg_database TO some_user;
+```
+
+MySQL:
+
+```mysql
+GRANT SHOW DATABASES ON *.* TO some_user;
+```
+
+The mechanisms in this guide should help you tackle different database
+deployment situations. Good luck!
@@ -55,10 +55,10 @@ const spotifyAuthApiRef = createApiRef<OAuthApi>({
 Sam realizes that Spotify auth might be useful to others, and that it would be
 more convenient if it was a part of the Backstage Core. After submitting and
 merging a Pull Request with the additions to the
-`@backstage/plugin-auth-backend` and `@backstage/core` packages, Spotify auth is
-now available for everyone to use. Since the Backstage Core team also adds it to
-the public demo server, Sam can now get rid of it in the local setup and rely on
-the shared development auth providers instead.
+`@backstage/plugin-auth-backend` and `@backstage/core-plugin-api` packages,
+Spotify auth is now available for everyone to use. Since the Backstage Core team
+also adds it to the public demo server, Sam can now get rid of it in the local
+setup and rely on the shared development auth providers instead.

 The only thing left now is making sure that users of the plugin provide Spotify
 auth in the app. Sam ensures this by adding `spotifyAuthApiRef` to the plugin's
@@ -70,7 +70,7 @@ README.

 This plugin requires the following APIs to function:

- `spotifyAuthApiRef` from `@backstage/core@^1.1.0`
+- `spotifyAuthApiRef` from `@@backstage/core-plugin-api@^1.1.0`
 ```

 # 3. The Catalog Awakens
@@ -0,0 +1,135 @@
+---
+id: migrating-away-from-core
+title: Migrating away from @backstage/core
+description: Guide on how to migrate to the new Backstage core libraries.
+---
+
+The `@backstage/core` package has been split into three separate packages,
+`@backstage/core-app-api`, `@backstage/core-plugin-api`, and
+`@backstage/core-components`. For more information about the reasoning behind
+this change and the naming of the packages, see the
+[original RFC](https://github.com/backstage/backstage/issues/4872) and
+[initial PR](https://github.com/backstage/backstage/pull/5825).
+
+The main purpose of the split is to make plugins more decoupled from the app,
+and open up for the possibility of combining plugins using many different
+versions of the core libraries. This should significantly reduce the maintenance
+burden on plugin authors, as well as reduce the impact of breaking changes in
+the core APIs.
+
+## Migration
+
+At a high level the migration is done by simply replacing usages of
+`@backstage/core` with one or more of the three new core libraries. There are a
+few breaking changes in the new packages that are listed below, but for most
+plugins the migration is a simple replacement. In order to make the migration as
+smooth as possible we provide a collection of tools to automate the majority of
+the migration effort.
+
+Below is a list of steps that should get most projects completely migrated, the
+order of the steps is a recommendation but not required, so don't worry if you
+need to go back to previous steps to fix things.
+
+### Step 1 - Run codemod
+
+The first step is to run
+[`@backstage/codemods`](https://www.npmjs.com/package/@backstage/codemods)
+across your project. This will automatically convert all module imports in your
+source code to use one of the three new core packages instead. For example, the
+following change might occur:
+
+```diff
+-import { useApi, configApiRef, InfoCard } from '@backstage/core';
+import { useApi, configApiRef } from '@backstage/core-plugin-api';
+import { InfoCard } from '@backstage/core-components';
+```
+
+In a typical app created with `@backstage/create-app`, you would run the
+following:
+
+```shell
+npx @backstage/codemods apply core-imports packages plugins
+```
+
+The last two arguments, `packages` and `plugins`, are the folders that the
+codemod should be applied to. Add or remove folders as needed for your project.
+
+The codemod might fail for some files because of the missing `IconKey` type in
+any of the new packages. This is one of the few breaking changes. To fix, remove
+any `IconKey` imports and replace usages of it with the `string` type, see the
+breaking changes section below for details. Once usages of `IconKey` type have
+been removed, you can re-run the codemod for those files.
+
+Note that while the codemod tries to stick to using the existing formatting in
+your project, it doesn't always manage to do that. If you're using `prettier` to
+format the code in your project, it's best to run `prettier --write` on any
+files that were changed by the codemod.
+
+### Step 2 - Update dependencies
+
+The next step is to update dependencies in your `package.json` files. Any
+package that currently depends on `@backstage/core` will need to have it
+replaced by one or more of the new packages. The app package should have all
+three packages added to `dependencies`, while for plugins and additional non-app
+packages, the `@backstage/core-plugin-api` and `@backstage/core-components`
+packages should be added to the set of regular `dependencies`, and
+`@backstage/core-app-api` should be added to `devDependencies` for usage in
+tests.
+
+A tool that can help out with step is the `plugin:diff` command from the
+`@backstage/cli`, it will compare your plugin to the base plugin template and
+suggest changes where the plugin deviates. A quick way to get this step done if
+you have up-to-date project is to run the following in the project root:
+
+```bash
+# The --yes flag causes all suggested changes to be accepted automatically
+yarn diff --yes
+```
+
+If you do not have the `diff` command set up in `package.json`, you can also
+manually execute the following in each plugin folder:
+
+```bash
+yarn backstage-cli plugin:diff --yes
+```
+
+### Step 3 - Manual review
+
+At this point your app is either completely or very close to being migrated. Run
+type checks with `yarn tsc` to check if you hit any of the breaking changes
+below or if there are any other things to fix. It can also be worthwhile
+searching for occurrences of `@backstage/core` in the codebase, as that might
+find usages in for example `jest` mock calls, which aren't handled by the
+codemod.
+
+As a final step you'll want to boot up the app and take it through any regular
+verification step that you have set up for your project. Don't hesitate to open
+a GitHub issue, PR, or reach out on Discord if you hit any snags, or if there
+are any additional steps or hints that you think should be added to this guide!
+
+## Breaking Changes
+
+The following is a list of breaking changes between `@backstage/core` and the
+three new core packages. Not that this list may not be exhaustive depending on
+when you migrate your app, as new releases of the new core packages may bring
+further changes.
+
+### Removed `IconKey` type
+
+The `IconKey` type used to be a string union of all known keys used for the app
+icons available through `useApp().getSystemIcon(key)`. The type has been removed
+since the set of allowed icon keys is no longer constrained, and there is
+instead only a guarantee that the app provides a minimum set of icons, but can
+provide any icons it wants beyond that. Migration is done by simply replacing
+old usages by the `string` type.
+
+### Constrained `IconComponent` type
+
+The `IconComponent` type used to allow all of the props from the MUI `SvgIcon`.
+This encouraged some bad patterns in open source plugins such as applying colors
+to the icons, which in turn hurt the ability to replace the icons with custom
+ones. The `IconComponent` type, which is now exported from
+`@backstage/core-plugin-api`, now only accepts a `fontSize` prop used to set the
+size of the icon. The type is compatible with the MUI `SvgIcon`, but there may
+be situations where an icon needs an explicit cast to `IconComponent` in order
+to narrow the type.
@@ -72,7 +72,7 @@ Our first modification will be to extract information from the Identity API.

 ```tsx
 // Add identityApiRef to the list of imported from core
-import { identityApiRef, useApi } from '@backstage/core';
+import { identityApiRef, useApi } from '@backstage/core-plugin-api';
 ```

 3. Adjust the ExampleComponent from inline to block
@@ -137,20 +137,13 @@ changes, let's start by wiping this component clean.
 import React from 'react';
 import { useAsync } from 'react-use';
 import Alert from '@material-ui/lab/Alert';
-import {
-  Table,
-  TableColumn,
-  Progress,
-  githubAuthApiRef,
-  useApi,
-} from '@backstage/core';
+import { Table, TableColumn, Progress } from '@backstage/core-components';
+import { githubAuthApiRef, useApi } from '@backstage/core-plugin-api';
 import { graphql } from '@octokit/graphql';

-const ExampleFetchComponent = () => {
+export const ExampleFetchComponent = () => {
  return <div>Nothing to see yet</div>;
 };
-
-export default ExampleFetchComponent;
 ```

 3. Save that and ensure you see no errors. Comment out the unused imports if