Merge branch 'master' of github.com:spotify/backstage into mob/docusaurus-structure

This commit is contained in:
Ivan Shmidt
2020-08-13 14:55:25 +02:00
61 changed files with 3955 additions and 71 deletions
@@ -18,6 +18,8 @@ humans. However, the structure and semantics is the same in both cases.
- [Common to All Kinds: The Envelope](#common-to-all-kinds-the-envelope)
- [Common to All Kinds: The Metadata](#common-to-all-kinds-the-metadata)
- [Kind: Component](#kind-component)
- [Kind: Template](#kind-template)
- [Kind: API](#kind-api)
## Overall Shape Of An Entity
@@ -256,6 +258,8 @@ spec:
type: website
lifecycle: production
owner: artist-relations@example.com
implementsApis:
- artist-api
```
In addition to the [common envelope metadata](#common-to-all-kinds-the-metadata)
@@ -316,14 +320,22 @@ Apart from being a string, the software catalog leaves the format of this field
open to implementers to choose. Most commonly, it is set to the ID or email of a
group of people in an organizational structure.
### `spec.implementsApis` [optional]
Links APIs that are implemented by the component, e.g. `artist-api`. This field
is optional.
The software catalog expects a list of one or more strings that references the
names of other entities of the `kind` `API`.
## Kind: Template
Describes the following entity kind:
| Field | Value |
| -------------------- | ----------------------- |
| `apiVersion` | `backstage.io/v1alpha1` |
| `Kind: Templatekind` | `Template` |
| Field | Value |
| ------------ | ----------------------- |
| `apiVersion` | `backstage.io/v1alpha1` |
| `kind` | `Template` |
A Template describes a skeleton for use with the Scaffolder. It is used for
describing what templating library is supported, and also for documenting the
@@ -426,3 +438,76 @@ specify relative to the `template.yaml` definition.
This is also particularly useful when you have multiple template definitions in
the same repository but only a single `template.yaml` registered in backstage.
## Kind: API
Describes the following entity kind:
| Field | Value |
| ------------ | ----------------------- |
| `apiVersion` | `backstage.io/v1alpha1` |
| `kind` | `API` |
An API describes an interface that can be exposed by a component. The API can be
defined in different formats, like [OpenAPI](https://swagger.io/specification/),
[AsyncAPI](https://www.asyncapi.com/docs/specifications/latest/),
[gRPC](https://developers.google.com/protocol-buffers), or other formats.
Descriptor files for this kind may look as follows.
```yaml
apiVersion: backstage.io/v1alpha1
kind: API
metadata:
name: artist-api
description: Retrieve artist details
spec:
type: openapi
definition: |
openapi: "3.0.0"
info:
version: 1.0.0
title: Artist API
license:
name: MIT
servers:
- url: http://artist.spotify.net/v1
paths:
/artists:
get:
summary: List all artists
...
```
In addition to the [common envelope metadata](#common-to-all-kinds-the-metadata)
shape, this kind has the following structure.
### `apiVersion` and `kind` [required]
Exactly equal to `backstage.io/v1alpha1` and `API`, respectively.
### `spec.type` [required]
The type of the API definition as a string, e.g. `openapi`. This field is
required.
The software catalog accepts any type value, but an organisation should take
great care to establish a proper taxonomy for these. Tools including Backstage
itself may read this field and behave differently depending on its value. For
example, an OpenAPI type API may be displayed using an OpenAPI viewer tooling in
the Backstage interface.
The current set of well-known and common values for this field is:
- `openapi` - An API definition in YAML or JSON format based on the
[OpenAPI](https://swagger.io/specification/) version 2 or version 3 spec.
- `asyncapi` - An API definition based on the
[AsyncAPI](https://www.asyncapi.com/docs/specifications/latest/) spec.
- `grpc` - An API definition based on
[Protocol Buffers](https://developers.google.com/protocol-buffers) to use with
[gRPC](https://grpc.io/).
### `spec.definition` [required]
The definition of the API, based on the format defined by `spec.type`. This
field is required.
+9 -2
View File
@@ -28,6 +28,13 @@ More specifically, the Service Catalog enables two main use-cases:
2. Makes all the software in your company, and who owns it, discoverable. No
more orphan software hiding in the dark corners of your software ecosystem.
## Getting Started
The Software Catalog is available to browse on the start page at `/`. If you've
followed [Installing in your Backstage App](./installation.md) in your separate
App or [Getting Started with Backstage](../../getting-started) for this repo,
you should be able to browse the catalog at `http://localhost:3000`.
![](../../assets/software-catalog/service-catalog-home.png)
## Adding components to the catalog
@@ -45,7 +52,7 @@ There are 3 ways to add components to the catalog:
### Manually register components
Users can register new components by going to `/create` and clicking the
**REGSITER EXISTING COMPONENT** button:
**REGISTER EXISTING COMPONENT** button:
![](bsc-register-1.png)
@@ -81,7 +88,7 @@ them, and do so using their normal Git workflow.
Once the change has been merged, Backstage will automatically show the updated
metadata in the service catalog after a short while.
## Finding software in the catlog
## Finding software in the catalog
By default the service catalog shows components owned by the team of the logged
in user. But you can also switch to _All_ to see all the components across your
@@ -0,0 +1,193 @@
# Installing in your Backstage App
The catalog plugin comes in two packages, `@backstage/plugin-catalog` and
`@backstage/plugin-catalog-backend`. Each has their own installation steps,
outlined below.
## Installing @backstage/plugin-catalog
> **Note that if you used `npx @backstage/create-app`, the plugin may already be
> present**
The catalog frontend plugin should be installed in your `app` package, which is
created as a part of `@backstage/create-app`. To install the package, run:
```bash
cd packages/app
yarn add @backstage/plugin-catalog
```
Make sure the version of `@backstage/plugin-catalog` matches the version of
other `@backstage` packages. You can update it in `packages/app/package.json` if
it doesn't.
### Adding the Plugin to your `packages/app`
Add the following entry to the head of your `packages/app/src/plugins.ts`:
```ts
export { plugin as CatalogPlugin } from '@backstage/plugin-catalog';
```
Add the following to your `packages/app/src/apis.ts`:
```ts
import { catalogApiRef, CatalogClient } from '@backstage/plugin-catalog';
// Inside the ApiRegistry builder function ...
builder.add(
catalogApiRef,
new CatalogClient({
apiOrigin: backendUrl,
basePath: '/catalog',
}),
);
```
Where `backendUrl` is the `backend.baseUrl` from config, i.e.
`const backendUrl = config.getString('backend.baseUrl')`.
The catalog components depend on a number of other
[Utility APIs](/docs/api/utility-apis.md) to function, including at least the
`ErrorApi` and `StorageApi`. You can find an example of how to install these in
your app
[here](https://github.com/spotify/backstage/blob/61c3a7e5b750dc7c059ef16b188594d31b2c04c2/packages/app/src/apis.ts#L80).
## Gotchas that we will fix
Since the catalog plugin currently ships with a sentry plugin `InfoCard`
installed by default, you'll need to set `sentry.organization` in your
`app-yaml.yaml`. For example:
```yaml
sentry:
organization: Acme Corporation
```
If you've created an app with an older version of `@backstage/create-app` or
`@backstage/cli create-app`, be sure to remove the Welcome plugin from the app,
as that will conflict with the catalog routes.
## Installing @backstage/plugin-catalog-backend
> **Note that if you used `npx @backstage/create-app`, the plugin may already be
> present**
The catalog backend should be installed in your `backend` package, which is
created as a part of `@backstage/create-app`. To install the package, run:
```bash
cd packages/backend
yarn add @backstage/plugin-catalog-backend
```
Make sure the version of `@backstage/plugin-catalog-backend` matches the version
of other `@backstage` packages. You can update it in
`packages/backend/package.json` if it doesn't.
### Adding the Plugin to your `packages/backend`
You'll need to add the plugin to the `backend`'s router. You can do this by
creating a file called `packages/backend/src/plugins/catalog.ts` with the
following contents to get you up and running quickly.
```ts
import {
createRouter,
DatabaseEntitiesCatalog,
DatabaseLocationsCatalog,
DatabaseManager,
HigherOrderOperations,
LocationReaders,
runPeriodically,
} from '@backstage/plugin-catalog-backend';
import { PluginEnvironment } from '../types';
import { useHotCleanup } from '@backstage/backend-common';
export default async function createPlugin({
logger,
database,
}: PluginEnvironment) {
const locationReader = new LocationReaders(logger);
const db = await DatabaseManager.createDatabase(database, { logger });
const entitiesCatalog = new DatabaseEntitiesCatalog(db);
const locationsCatalog = new DatabaseLocationsCatalog(db);
const higherOrderOperation = new HigherOrderOperations(
entitiesCatalog,
locationsCatalog,
locationReader,
logger,
);
useHotCleanup(
module,
runPeriodically(() => higherOrderOperation.refreshAllLocations(), 10000),
);
return await createRouter({
entitiesCatalog,
locationsCatalog,
higherOrderOperation,
logger,
});
}
```
Once the `catalog.ts` router setup file is in place, add the router to
`packages/backend/src/index.ts`:
```ts
import catalog from './plugins/catalog';
const catalogEnv = useHotMemoize(module, () => createEnv('catalog'));
const service = createServiceBuilder(module)
.loadConfig(configReader)
/** several different routers */
.addRouter('/catalog', await catalog(catalogEnv));
```
### Adding Entries to the Catalog
At this point the catalog backend is installed in your backend package, but you
will not have any entities loaded.
To get up and running and try out some templates quickly, you can add some of
our example templates through static configuration. Add the following to the
`catalog.locations` section in your `app-config.yaml`:
```yaml
catalog:
locations:
# Backstage Example Component
- type: github
target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/artist-lookup-component.yaml
- type: github
target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/playback-order-component.yaml
- type: github
target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/podcast-api-component.yaml
- type: github
target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/queue-proxy-component.yaml
- type: github
target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/searcher-component.yaml
- type: github
target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/playback-lib-component.yaml
- type: github
target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/www-artist-component.yaml
- type: github
target: https://github.com/spotify/backstage/blob/master/packages/catalog-model/examples/shuffle-api-component.yaml
```
### Running the Backend
Finally, start up the backend with the new configuration:
```bash
cd packages/backend
yarn start
```
If you've also set up the frontend plugin, so you should be ready to go browse
the catalog at [localhost:3000](http://localhost:3000) now!
@@ -8,7 +8,7 @@ software and resources leads to a better Backstage experience.
_This description originates from
[this RFC](https://github.com/spotify/backstage/issues/390). Note that some of
the concpets are not yet supported in Backstage._
the concepts are not yet supported in Backstage._
## Concepts
+5 -4
View File
@@ -14,9 +14,10 @@ like GitHub.
### Getting Started
The Software Templates are available under `/create`, and if you've followed
[Getting Started with Backstage](../../getting-started), you should be able to
reach `http://localhost:3000/create`.
The Software Templates are available under `/create`. If you've followed
[Installing in your Backstage App](./installation.md) in your separate App or
[Getting Started with Backstage](../../getting-started) for this repo, you
should be able to reach `http://localhost:3000/create`.
You should get something that looks similar to this:
@@ -34,7 +35,7 @@ internally.
After filling in these variables, you'll get some more fields to fill out which
are required for backstage usage. The owner, which is a `user` in the backstage
system, and the `storePath` which right now must be a Github Organisation and a
non-existing github repository name in the format `organistaion/reponame`.
non-existing github repository name in the format `organisation/reponame`.
![Enter backstage vars](../../assets/software-templates/template-picked-2.png)
@@ -0,0 +1,194 @@
# Installing in your Backstage App
The scaffolder plugin comes in two packages, `@backstage/plugin-scaffolder` and
`@backstage/plugin-scaffolder-backend`. Each has their own installation steps,
outlined below.
The Scaffolder plugin also depends on the Software Catalog. Instructions for how
to set that up can be found [here](../software-catalog/installation.md).
## Installing @backstage/plugin-scaffolder
> **Note that if you used `npx @backstage/create-app`, the plugin may already be
> present**
The scaffolder frontend plugin should be installed in your `app` package, which
is created as a part of `@backstage/create-app`. To install the package, run:
```bash
cd packages/app
yarn add @backstage/plugin-scaffolder
```
Make sure the version of `@backstage/plugin-scaffolder` matches the version of
other `@backstage` packages. You can update it in `packages/app/package.json` if
it doesn't.
### Adding the Plugin to your `packages/app`
Add the following entry to the head of your `packages/app/src/plugins.ts`:
```ts
export { plugin as ScaffolderPlugin } from '@backstage/plugin-scaffolder';
```
Add the following to your `packages/app/src/apis.ts`:
```ts
import { scaffolderApiRef, ScaffolderApi } from '@backstage/plugin-scaffolder';
// Inside the ApiRegistry builder function ...
builder.add(
scaffolderApiRef,
new ScaffolderApi({
apiOrigin: backendUrl,
basePath: '/scaffolder/v1',
}),
);
```
Where `backendUrl` is the `backend.baseUrl` from config, i.e.
`const backendUrl = config.getString('backend.baseUrl')`.
This is all that is needed for the frontend part of the Scaffolder plugin to
work!
## Installing @backstage/plugin-scaffolder-backend
> **Note that if you used `npx @backstage/create-app`, the plugin may already be
> present**
The scaffolder backend should be installed in your `backend` package, which is
created as a part of `@backstage/create-app`. To install the package, run:
```bash
cd packages/backend
yarn add @backstage/plugin-scaffolder-backend
```
Make sure the version of `@backstage/plugin-scaffolder-backend` matches the
version of other `@backstage` packages. You can update it in
`packages/backend/package.json` if it doesn't.
### Adding the Plugin to your `packages/backend`
You'll need to add the plugin to the `backend`'s router. You can do this by
creating a file called `packages/backend/src/plugins/scaffolder.ts` with the
following contents to get you up and running quickly.
```ts
import {
CookieCutter,
createRouter,
FilePreparer,
GithubPreparer,
Preparers,
GithubPublisher,
CreateReactAppTemplater,
Templaters,
} from '@backstage/plugin-scaffolder-backend';
import { Octokit } from '@octokit/rest';
import type { PluginEnvironment } from '../types';
import Docker from 'dockerode';
export default async function createPlugin({ logger }: PluginEnvironment) {
const cookiecutterTemplater = new CookieCutter();
const craTemplater = new CreateReactAppTemplater();
const templaters = new Templaters();
// Register default templaters
templaters.register('cookiecutter', cookiecutterTemplater);
templaters.register('cra', craTemplater);
const filePreparer = new FilePreparer();
const githubPreparer = new GithubPreparer();
const preparers = new Preparers();
// Register default preparers
preparers.register('file', filePreparer);
preparers.register('github', githubPreparer);
// Create Github client with your access token from environment variables
const githubClient = new Octokit({ auth: process.env.GITHUB_ACCESS_TOKEN });
const publisher = new GithubPublisher({ client: githubClient });
const dockerClient = new Docker();
return await createRouter({
preparers,
templaters,
publisher,
logger,
dockerClient,
});
}
```
Once the `scaffolder.ts` router setup file is in place, add the router to
`packages/backend/src/index.ts`:
```ts
import scaffolder from './plugins/scaffolder';
const scaffolderEnv = useHotMemoize(module, () => createEnv('scaffolder'));
const service = createServiceBuilder(module)
.loadConfig(configReader)
/** several different routers */
.addRouter('/scaffolder', await scaffolder(scaffolderEnv));
```
### Adding Templates
At this point the scaffolder backend is installed in your backend package, but
you will not have any templates available to use. These need to be added to the
software catalog, as they are represented as entities of kind
[Template](/docs/features/software-catalog/descriptor-format.md#kind-template).
You can find out more about adding templates [here](./adding-templates.md).
To get up and running and try out some templates quickly, you can add some of
our example templates through static configuration. Add the following to the
`catalog.locations` section in your `app-config.yaml`:
```yaml
catalog:
locations:
# Backstage Example Templates
- type: github
target: https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/sample-templates/react-ssr-template/template.yaml
- type: github
target: https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/sample-templates/springboot-grpc-template/template.yaml
- type: github
target: https://github.com/spotify/backstage/blob/master/plugins/scaffolder-backend/sample-templates/create-react-app/template.yaml
- type: github
target: https://github.com/spotify/cookiecutter-golang/blob/master/template.yaml
```
### Runtime Dependencies
For the scaffolder backend plugin to function, it needs a GitHub access token,
and access to a running Docker daemon. You can create a GitHub access token
[here](https://github.com/settings/tokens/new), select `repo` scope only. Full
docs on creating private GitHub access tokens is available
[here](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token).
Note that the need for private GitHub access tokens will be replaced with GitHub
Apps integration further down the line.
> **Right now it is only possible to scaffold repositories inside GitHub
> organizations, and not under personal accounts.**
The GitHub access token is passed along using the `GITHUB_ACCESS_TOKEN`
environment variable.
### Running the Backend
Finally, make sure you have a local Docker daemon running, and start up the
backend with the new configuration:
```bash
cd packages/backend
GITHUB_ACCESS_TOKEN=<token> yarn start
```
If you've also set up the frontend plugin, so you should be ready to go browse
the templates at [localhost:3000/create](http://localhost:3000/create) now!
+7 -7
View File
@@ -15,15 +15,15 @@ to create a streamlined development environment from end to end.
Out of the box, Backstage includes:
- [Backstage Service Catalog](https://github.com/spotify/backstage/blob/master/docs/features/software-catalog/index.md)
for managing all your software (microservices, libraries, data pipelines,
- [Backstage Service Catalog](/docs/features/software-catalog/index.md) for
managing all your software (microservices, libraries, data pipelines,
websites, ML models, etc.)
- [Backstage Software Templates](https://github.com/spotify/backstage/blob/master/docs/features/software-templates/index.md)
for quickly spinning up new projects and standardizing your tooling with your
- [Backstage Software Templates](/docs/features/software-templates/index.md) for
quickly spinning up new projects and standardizing your tooling with your
organizations best practices
- [Backstage TechDocs](https://github.com/spotify/backstage/tree/master/docs/features/techdocs)
for making it easy to create, maintain, find, and use technical documentation,
using a "docs like code" approach
- [Backstage TechDocs](/docs/features/techdocs) for making it easy to create,
maintain, find, and use technical documentation, using a "docs like code"
approach
- Plus, a growing ecosystem of
[open source plugins](https://github.com/spotify/backstage/tree/master/plugins)
that further expand Backstages customizability and functionality
+19 -4
View File
@@ -17,12 +17,21 @@
const { resolve: resolvePath, dirname } = require('path');
const fs = require('fs-extra');
const fetch = require('node-fetch');
const recursive = require('recursive-readdir');
const projectRoot = resolvePath(__dirname, '..');
async function verifyUrl(basePath, url) {
// Avoid having absolute URL links within docs/, so that links work on the site
if (
url.match(
/https:\/\/github.com\/spotify\/backstage\/(tree|blob)\/master\/docs\//,
) &&
basePath.match(/^(?:docs|microsite)\//)
) {
return { url, basePath, problem: 'absolute' };
}
url = url.replace(/#.*$/, '');
url = url.replace(
/https:\/\/github.com\/spotify\/backstage\/(tree|blob)\/master/,
@@ -39,7 +48,7 @@ async function verifyUrl(basePath, url) {
: resolvePath(dirname(resolvePath(projectRoot, basePath)), url);
const exists = await fs.pathExists(path);
if (!exists) {
return { url, basePath };
return { url, basePath, problem: 'missing' };
}
}
@@ -81,8 +90,14 @@ async function main() {
if (badUrls.length) {
console.log(`Found ${badUrls.length} bad links within repo`);
for (const { url, basePath } of badUrls) {
console.error(`Unable to reach ${url}, linked from ${basePath}`);
for (const { url, basePath, problem } of badUrls) {
if (problem === 'missing') {
console.error(`Unable to reach ${url}, linked from ${basePath}`);
} else if (problem === 'absolute') {
console.error(`Link to docs/ should be replaced by a relative URL`);
console.error(` From: ${basePath}`);
console.error(` To: ${url}`);
}
}
process.exit(1);
}