Scaffolder Documentation (#1616)

* docs(scaffolder): Added some reference for the Template kind in the service catalog * docs(scaffolder): adding a basic readme for the scaffolder * docs(scaffolder): more small fixes * chore(scaffolder): removing unused routes in the scaffolder * docs(scaffolder): fixing linking to the template entity * docs(scaffolder): fixing indentation * docs(scaffolder): rewrite a little * chore(scaffolder): Added some more documentation sections * docs(scaffolder): added some documentation about loading your own templates * docs(scaffolder): created some great documentation. make docs great again * docs(scaffolder): added some more info about creating your own templater * chore(scaffolder): finshing of create your own templater docs * docs(scaffolder): updating the last of the preparr * docs(scaffolder): think docs are great for scaffolder again * chore(scaffolder): fixing issue with publisher base mistyping * docs(scaffolder): creating some how to use documentation * docs(scaffolder): more user docs for the scaffolder * docs(scaffolder): fix header * docs(scaffolder): fixing * docs(scaffolder): shuffle around the documenation
2020-07-15 16:32:44 +02:00
parent c4e02d2057
commit a14dae9b10
22 changed files with 707 additions and 18 deletions
@@ -311,3 +311,114 @@ component, but there will always be one ultimate owner.
 Apart from being a string, the software catalog leaves the format of this field
 open to implementers to choose. Most commonly, it is set to the ID or email of a
 group of people in an organizational structure.
+
+##
+
+Describes the following entity kind:
+
+| Field                | Value                   |
+| -------------------- | ----------------------- |
+| `apiVersion`         | `backstage.io/v1alpha1` |
+| `Kind: Templatekind` | `Template`              |
+
+A Template describes a skeleton for use with the Scaffolder. It is used for
+describing what templating library is supported, and also for documenting the
+variables that the template requires using
+[JSON Forms Schema](https://jsonforms.io/).
+
+Descriptor files for this kind may look as follows.
+
+```yaml
+apiVersion: backstage.io/v1alpha1
+kind: Template
+metadata:
+  name: react-ssr-template
+  title: React SSR Template
+  description:
+    Next.js application skeleton for creating isomorphic web applications.
+  tags:
+    - Recommended
+    - React
+spec:
+  owner: web@example.com
+  templater: cookiecutter
+  type: website
+  path: '.'
+  schema:
+    required:
+      - component_id
+      - description
+    properties:
+      component_id:
+        title: Name
+        type: string
+        description: Unique name of the component
+      description:
+        title: Description
+        type: string
+        description: Description of the component
+```
+
+In addition to the [common envelope metadata](#common-to-all-kinds-the-metadata)
+shape, this kind has the following structure.
+
+### `apiVersion` and `kind` [required]
+
+Exactly equal to `backstage.io/v1alpha1` and `Template`, respectively.
+
+### `metadata.title` [required]
+
+The nice display name for the template as a string, e.g. `React SSR Template`.
+This field is required as is used to reference the template to the user instead
+of the `metadata.name` field.
+
+### `metadata.tags` [optional]
+
+A list of strings that can be associated with the template, e.g.
+`['Recommended', 'React']`.
+
+This list will also be used in the frontend to display to the user so you can
+potentially search and group templates by these tags.
+
+### `spec.type` [optional]
+
+The type of component as a string, e.g. `website`. This field is optional but
+recommended.
+
+The software catalog accepts any type value, but an organisation should take
+great care to establish a proper taxonomy for these. Tools including Backstage
+itself may read this field and behave differently depending on its value. For
+example, a website type component may present tooling in the Backstage interface
+that is specific to just websites.
+
+The current set of well-known and common values for this field is:
+
+- `service` - a backend service, typically exposing an API
+- `website` - a website
+- `library` - a software library, such as an NPM module or a Java library
+
+### `spec.templater` [required]
+
+The templating library that is supported by the template skeleton as a string,
+e.g `cookiecutter`.
+
+Different skeletons will use different templating syntax, so it's common that
+the template will need to be run with a particular piece of software.
+
+This key will be used to identify the correct templater which is registered into
+the `TemplatersBuilder`.
+
+The values which are available by default are:
+
+- `cookiecutter` - [cookiecutter](https://github.com/cookiecutter/cookiecutter).
+
+### `spec.path` [optional]
+
+The string location where the templater should be run if it is not on the same
+level as the `template.yaml` definition, e.g. `./cookiecutter/skeleton`.
+
+This will set the `cwd` when running the templater to the folder path that you
+specify relative to the `template.yaml` definition.
+
+This is also particularly useful when you have multiple template definitions in
+the same repository but only a single `template.yaml` registered in backstage.
@@ -0,0 +1,102 @@
+# Adding your own Templates
+
+Templates are stored in the **Service Catalog** under a kind `Template`. The
+minimum that the a template skeleton needs is a `template.yaml` but it would be
+good to also have some files in there that can be templated in.
+
+A simple `template.yaml` definition might look something like this:
+
+```yaml
+apiVersion: backstage.io/v1alpha1
+kind: Template
+metadata:
+	# unique name per namespace for the template
+  name: react-ssr-template
+	# title of the template
+  title: React SSR Template
+	# a description of the template
+  description: Next.js application skeleton for creating isomorphic web applications.
+  # some tags to display in the frontend
+	tags:
+    - Recommended
+    - React
+spec:
+  # which templater key to use in the templaters builder
+  templater: cookiecutter
+	# what does this template create
+  type: website
+	# if the template is not in the current directory where this definition is kept then specfiy
+  path: './template'
+	# the schema for the form which is displayed in the frontend.
+  # should follow JSON schema for forms: https://jsonforms.io/
+  schema:
+    required:
+      - component_id
+      - description
+    properties:
+      component_id:
+        title: Name
+        type: string
+        description: Unique name of the component
+      description:
+        title:  Description
+        type: string
+        description: Description of the component
+```
+
+[Template Entity](../software-catalog/descriptor-format.md#kind-template)
+contains more information about the required fields.
+
+Once we have a `template.yaml` ready, we can then add it to the service catalog
+for use by the scaffolder.
+
+Currently the catalog supports loading definitions from Github + Local Files. To
+load from other places, not only will there need to be another preparer, but the
+support to load the location will also need to be added to the Catalog.
+
+For loading from a file the following command should work when the backend is
+running:
+
+```sh
+curl \
+  --location \
+  --request POST 'localhost:7000/catalog/locations' \
+  --header 'Content-Type: application/json' \
+  --data-raw "{\"type\": \"file\", \"target\": \"${YOUR PATH HERE}/template.yaml\"}"
+```
+
+If loading from a git location, you can run the following
+
+```sh
+curl \
+  --location \
+  --request POST 'localhost:7000/catalog/locations' \
+  --header 'Content-Type: application/json' \
+  --data-raw "{\"type\": \"github\", \"target\": \"https://${YOUR GITHUB REPO}blob/master/${PATH TO FOLDER}/template.yaml\"}"
+```
+
+This should then have added the catalog, and also should now be listed under the
+create page at http://localhost:3000/create.
+
+Alternatively, if you want to get setup with some mock templates that are
+already provided for you, you can run the following to load those templates:
+
+```
+yarn lerna run mock-data
+```
+
+The `type` field which is chosen in the request to add the `template.yaml` to
+the Service Catalog here, will be come the `PreparerKey` which will be used to
+select the `Preparer` when creating a job.
+
+### Adding form values in the Scaffolder Wizard
+
+The `spec.schema` property in the
+[Template Entity](../software-catalog/descriptor-format.md#kind-template) is a
+`yaml` version of the JSON Form Schema standard.
+
+Here you can define the key/values and then the wizard will convert this to a
+form for the user to fill in when your template is selected.
+
+You can find out much more about the standard and how to use it here:
+https://jsonforms.io
@@ -0,0 +1,97 @@
+# Create your own Preparer
+
+Preparers are responsible for reading the location of the definition of a
+[Template Entity](../../software-catalog/descriptor-format.md#kind-template) and
+making a temporary folder with the contents of the selected skeleton.
+
+Currently, we provide two different providers that can parse two different
+location protocols:
+
+- `file://`
+- `github://`
+
+These two are added to the `PreparersBuilder` and then passed into the
+`createRouter` function of the `@spotify/plugin-scaffolder-backend`
+
+An full example backend can be found
+[here](https://github.com/spotify/backstage/blob/d91c10f654475a60829fa33a5c81018e517a319a/packages/backend/src/plugins/scaffolder.ts),
+but it looks something like the following
+
+```ts
+import {
+  createRouter,
+  FilePreparer,
+  GithubPreparer,
+  Preparers,
+} from '@backstage/plugin-scaffolder-backend';
+import type { PluginEnvironment } from '../types';
+
+export default async function createPlugin({ logger }: PluginEnvironment) {
+  const preparers = new Preparers();
+
+  const filePreparer = new FilePreparer();
+  const githubPreparer = new GithubPreparer();
+
+  preparers.register('file', filePreparer);
+  preparers.register('github', githubPreparer);
+
+  return await createRouter({
+    preparers,
+    templaters,
+    logger,
+  });
+}
+```
+
+As you can see in the above code, a `PreparerBuilder` is created, and then two
+of the `preparers` are registered with the different protocols that they accept.
+
+The `protocol` is set on the
+[Template Entity](../../software-catalog/descriptor-format.md#kind-template)
+when added to the service catalog. You can see more about this `PreparerKey`
+here in [Register your own template](../adding-templates.md)
+
+**note:** Currently the catalog supports loading definitions from Github + Local
+Files, which translate into the two `PreparerKeys` `file` and `github`. To load
+from other places, not only will there need to be another preparer, but the
+support to load the location will also need to be added to the Catalog.
+
+### Creating your own Preparer to add to the `PreparerBuilder`
+
+All preparers need to implement the `PreparerBase` type.
+
+That type looks like the following:
+
+```ts
+export type PreparerBase = {
+  prepare(
+    template: TemplateEntityV1alpha1,
+    opts: { logger: Logger },
+  ): Promise<string>;
+};
+```
+
+The `prepare` function will be given the
+[Template Entity](../../software-catalog/descriptor-format.md#kind-template)
+along with the source of where the `template.yaml` was loaded from under the
+`metedata.annotations.managed-by-location` property.
+
+Now it's up to you to implement a function which can go and fetch the skeleton
+and put the contents into a temporary directory and return that directory path.
+
+Some good examples exist here:
+
+- https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/src/scaffolder/stages/prepare/file.ts
+- https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/src/scaffolder/stages/prepare/github.ts
+
+### Registerinng your own Preparer
+
+You can register the preparer that you have created with the `PreparerBuilder`
+by using the `PreparerKey` from the Catalog, for example like this:
+
+```ts
+const preparers = new Preparers();
+preparers.register('gcs', new GoogleCloudStoragePreparer());
+```
+
+And then pass this in to the `createRouter` function.
@@ -0,0 +1,92 @@
+# Create your own Publisher
+
+Publishers are responsible for pushing and storing the templated skeleton after
+the values have been templated by the `Templater`. See
+[Create your own templater](./create-your-own-templater.md) for more info.
+
+They recieve a directory or location where the templater has sucessfully run on,
+and is now ready to store somewhere. They also get given some other options
+which are sent from the frontend, such as the `storePath` which is a string of
+where the frontend thinks we should save this templated folder.
+
+Currently we provide the following `publishers`:
+
+- `github`
+
+This publisher is passed through to the `createRouter` function of the
+`@spotify/plugin-scaffolder-backend`. Currently only one publisher is supported,
+but PR's are always welcome.
+
+An full example backend can be found
+[here](https://github.com/spotify/backstage/blob/d91c10f654475a60829fa33a5c81018e517a319a/packages/backend/src/plugins/scaffolder.ts),
+but it looks something like the following
+
+```ts
+import {
+  createRouter,
+  GithubPublisher,
+} from '@backstage/plugin-scaffolder-backend';
+import { Octokit } from '@octokit/rest';
+import type { PluginEnvironment } from '../types';
+
+export default async function createPlugin({ logger }: PluginEnvironment) {
+  const githubClient = new Octokit({ auth: process.env.GITHUB_ACCESS_TOKEN });
+  const publisher = new GithubPublisher({ client: githubClient });
+
+  return await createRouter({
+    publisher,
+    logger,
+  });
+}
+```
+
+The publisher will always be called with the location from the selected
+`Preparer`.
+
+### Create your own Publisher and register it with the Scaffolder
+
+All `publishers` need to implement the `PublisherBase` type.
+
+That type looks like the following:
+
+```ts
+export type PublisherBase = {
+  publish(opts: {
+    entity: TemplateEntityV1alpha1;
+    values: RequiredTemplateValues & Record<string, JsonValue>;
+    directory: string;
+  }): Promise<{ remoteUrl: string }>;
+};
+```
+
+The `publisher` function will be called with an `options` object which contains
+the following:
+
+- `entity` - the
+  [Template Entity](../../software-catalog/descriptor-format.md#kind-template)
+  which is currently being scaffolded
+- `values` - a json object which will resemble the `spec.schema` from the
+  [Template Entity](../../software-catalog/descriptor-format.md#kind-template)
+  which is defined here under spec.schema`. More info can be found here
+  [Register your own template](../adding-templates.md#adding-form-values-in-the-scaffolder-wizard)
+- `directory` - a string containing the returned path from the `templater`. See
+  more information here in
+  [Create your own templater](./create-your-own-templater.md)
+
+Now it's up to you to implement the `publish` function and return
+`{ remoteUrl: string }` which can be used to identify the finished product.
+
+Some good examples exist here:
+
+- https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/src/scaffolder/stages/publish/github.ts
+
+### Registering your own Publisher
+
+Currently, we only support one `publisher` (PR's welcome), but you can register
+any single `publisher` with the `createRouter`.
+
+```ts
+return await createRouter({
+  publisher: new MyGitlabPublisher(),
+});
+```
@@ -0,0 +1,146 @@
+# Creating your own Templater
+
+Templaters are responsible for taking the directory path for the skeleton
+returned by the preparers, and then executing the templating command on top of
+the file and returning the completed template path. This may or may not be the
+same directory as the input directory.
+
+They also recieve additional values from the frontend, which can be used to
+interpolate into the skeleton files.
+
+Currently we provide the following templaters:
+
+- `cookiecutter`
+
+This templater is added the `TemplaterBuilder` and then passed into the
+`createRouter` function of the `@spotify/plugin-scaffolder-backend`
+
+An full example backend can be found
+[here](https://github.com/spotify/backstage/blob/d91c10f654475a60829fa33a5c81018e517a319a/packages/backend/src/plugins/scaffolder.ts),
+but it looks something like the following
+
+```ts
+import {
+  CookieCutter,
+  createRouter,
+  Templaters,
+} from '@backstage/plugin-scaffolder-backend';
+import type { PluginEnvironment } from '../types';
+
+export default async function createPlugin({ logger }: PluginEnvironment) {
+  const templaters = new Templaters();
+  const cookiecutterTemplater = new CookieCutter();
+  templaters.register('cookiecutter', cookiecutterTemplater);
+
+  return await createRouter({
+    templaters,
+  });
+}
+```
+
+As you can see in the above code a `TemplaterBuilder` is created and the default
+`cookiecutter` `templater` is registered under the key `cookicutter`.
+
+This `TemplaterKey` is used to select the correct templater from the
+`spec.templater` in the
+[Template Entity](../software-catalog/descriptor-format.md#kind-template).
+
+If you wish to add a new templater you'll need to register it with the
+`TemplaterBuilder`.
+
+### Creating your own Templater to add to the `TemplaterBuilder`
+
+All templaters need to implement the `TemplaterBase` type.
+
+That type looks like the following:
+
+```ts
+export type TemplaterRunOptions = {
+  directory: string;
+  values: RequiredTemplateValues & Record<string, JsonValue>;
+  logStream?: Writable;
+  dockerClient: Docker;
+};
+
+export type TemplaterBase = {
+  run(opts: TemplaterRunOptions): Promise<TemplaterRunResult>;
+};
+```
+
+The `run` function will be given a `TemplaterRunOptions` object which is as
+follows:
+
+- `directory`- the skeleton directory returned from the `Preparer`, more info at
+  [Create your own preparer](./create-your-own-preparer.md).
+- `values` - a json object which will resemble the `spec.schema` from the
+  [Template Entity](../software-catalog/descriptor-format.md#kind-template)
+  which is defined here under spec.schema`. More info can be found here
+  [Register your own template](./register-your-own-template.md#adding-form-values-in-the-scaffolder-wizard)
+- `logStream` - a stream that you can write to for displaying in the frontend.
+- `dockerClient` - a [dockerode](https://github.com/apocas/dockerode) client to
+  be able to run docker containers.
+
+_note_ currently the templaters that we provide are basically docker action
+containers that are run on top of the skeleton folder. This keeps dependencies
+to a minimal for running backstage scaffolder, but you don't /have/ to use
+docker. You could create your own templater that spins up an EC2 instance and
+downloads the folder and does everything using an AMI if you want. It's entirely
+up to you!
+
+Now it's up to you to implement the `run` function, and then return a
+`TemplaterRunResult` which is `{ resultDir: string }`.
+
+Some good examples exist here:
+
+- https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/src/scaffolder/stages/templater/cookiecutter.ts
+
+### Registering your own Templater
+
+If you try to process a
+[Template Entity](../../software-catalog/descriptor-format.md#kind-template)
+with a new `spec.templater` value, you'll need to register that with the
+`TemplaterBuilder`.
+
+For example let's say you have the following
+[Template Entity](../../software-catalog/descriptor-format.md#kind-template):
+
+```yaml
+apiVersion: backstage.io/v1alpha1
+kind: Template
+metadata:
+  name: react-ssr-template
+  title: React SSR Template
+  description:
+    Next.js application skeleton for creating isomorphic web applications.
+  tags:
+    - Recommended
+    - React
+spec:
+  owner: web@example.com
+  templater: handlebars
+  type: website
+  path: '.'
+  schema:
+    required:
+      - component_id
+      - description
+    properties:
+      component_id:
+        title: Name
+        type: string
+        description: Unique name of the component
+      description:
+        title: Description
+        type: string
+        description: Description of the component
+```
+
+You see that the `spec.templater` is set as `handlebars`, you'll need to
+register this with the `TemplaterBuilder` like so:
+
+```ts
+const templaters = new Templaters();
+templaters.register('handlebars', new HandlebarsTemplater());
+```
+
+And then pass this in to the `createRouter` function.
@@ -0,0 +1,87 @@
+## Extending the Scaffolder
+
+Welcome. Take a seat. You're at the Scaffolder Documentation.
+
+So - You wanna create stuff inside your company from some prebaked templates?
+You're at the right place.
+
+This guide is gonna take you through how the Scaffolder in Backstage works.
+We'll dive into some jargon and run through whats going on in the backend to be
+able to create these templates. There's also more guides that you might find
+useful at the bottom of this document. At it's core, theres 3 simple stages.
+
+1. Pick a skeleton
+2. Template some variables into the skeleton
+3. Send the templated skeleton somewhere
+
+These three steps are translated to the folllowing stages under the hood in the
+scaffolder that you will need to know:
+
+1. Prepare
+2. Template
+3. Publish
+
+Each of these steps can be configured for your own use case, but we provide some
+sensible defaults too.
+
+Lets dive a little deeper into these phases.
+
+### Glossary and Jargon
+
+**Preparer** - The preparer is responsible for fetching the skeleton code and
+placing it into a directory and then will return that directory. It is
+registered with the `Preparers` with a particular type, which is then used in
+the router to pick the correct `Preparer` to run for the `Template` entity.
+
+**Templater** - The templater is responsible for actually running the chosen
+templater on top of the previously returned temporary directory from the
+**Preprarer**. We advise making these docker containers as it can keep all
+dependencies, for example Cookiecutter, self contained and not a dependency on
+the host machine.
+
+**Publisher** - The publisher is responsible for taking the finished directory,
+and publishing it to a remote registry. This could be a Git repository or
+something similar. Right now, the scaffolder only supports one publishing method
+for the entire lifecycle, but it could be configured from the frontend and
+passed through to the scaffolder backend.
+
+### How it works
+
+The main of the heavy lifting is done in the
+[router.ts](https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/src/service/router.ts#L93)
+file in the `scaffolder-backend` plugin.
+
+There are 2 routes defined in the router. `POST /v1/jobs` and
+`GET /v1/job/:jobId`
+
+To create a scaffolding job, a JSON object containing the
+[Template Entity](../../software-catalog/descriptor-format.md#kind-template) +
+additional templating values must be posted as the post body.
+
+```js
+{
+  "template": {
+    "apiVersion": "backstage/v1alpha1",
+    "kind": "Template",
+    // more stuff here
+  },
+  "values": {
+    "component_id": "test",
+    "description": "somethingelse"
+  }
+}
+```
+
+The values should represent something that is valid with the `schema` part of
+the [Template Entity](../../software-catalog/descriptor-format.md#kind-template)
+
+Once that has been posted, a job will be setup with different stages. And the
+job processor will complete each stage before moving onto the next stage, whilst
+collecting logs and mutating the running job.
+
+Here's some futher reading that you might find useful:
+
+- [Adding your own Template](../adding-templates.md)
+- [Creating your own Templater](./create-your-own-templater.md)
+- [Creating your own Publisher](./create-your-own-publisher.md)
+- [Creating your own Preparer](./create-your-own-preparer.md)
@@ -0,0 +1,59 @@
+# Software Templates
+
+The Software Templates part of Backstage is a tool that can help you create
+Components inside Backstage. It by default has the ability to load skeletons of
+code, template in some variables and then publish the template to some location
+like GitHub.
+
+### Getting Started
+
+The Software Templates are available under `/create`, and if you've followed
+[Getting Started with Backstage](../../getting-started), you should be able to
+reach `http://localhost:3000/create`.
+
+You should get something that looks similar to this:
+
+![Create Image](./assets/create.png)
+
+### Choose a template
+
+When you select a template that you want to create, you'll be taken to the next
+page which may or may not look different for each template. Each template can
+ask for different input variables, and they are then passed to the templater
+internally.
+
+![Enter some variables](./assets/template-picked.png)
+
+After filling in these variables, you'll get some more fields to fill out which
+are required for backstage usage. The owner, which is a `user` in the backstage
+system, and the `storePath` which right now must be a Github Organisation and a
+non-existing github repository name in the format `organistaion/reponame`.
+
+![Enter backstage vars](./assets/template-picked-1.png)
+
+### Run!
+
+Once you've entered values and confirmed, you'll then get a modal with live
+progress of what is currently happening with the creation of your template.
+
+![Templating Running](./assets/running.png)
+
+It shouldn't take too long, and you'll have a success screen!
+
+![Templating Complete](./assets/complete.png)
+
+If it fails, you'll be able to click on each section to get the log from the
+step that failed which can be helpful to debug.
+
+![Templating failed](./assets/failed.png)
+
+### View Component in Catalog
+
+When it's been created you'll see the `View in Catalog` button, which will take
+you to the registered component in the catalog:
+
+![Catalog](./assets/go-to-catalog.png)
+
+And then you'll also be able to see it in the Catalog View table
+
+![Catalog](./assets/added-to-the-catalog-list.png)