paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 32 Packages Projects Releases Wiki Activity

Merge branch 'master' into github-installations-limit

Browse Source
This commit is contained in:
Brian Fletcher
2021-07-14 08:03:08 +01:00
committed by GitHub
parent 5fa0c32bff 474cb76c88
commit a5d112ef49
3877 changed files with 39360 additions and 53613 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plugins/backend-plugin.md
+40 -1
View File
@@ -99,7 +99,7 @@ import carmen from './plugins/carmen';
async function main() {
// ...
const carmenEnv = useHotMemoize(module, () => createEnv('carmen'));
apiRouter.use('/carmen', await carmen(badgesEnv));
apiRouter.use('/carmen', await carmen(carmenEnv));
```
After you start the backend (e.g. using `yarn start-backend` from the repo
@@ -111,3 +111,42 @@ curl localhost:7000/api/carmen/health
```
This should return `{"status":"ok"}` like before. Success!
## Making Use of a Database
The Backstage backend comes with a builtin facility for SQL database access.
Most plugins that have persistence needs will choose to make use of this
facility, so that Backstage operators can manage database needs uniformly.
As part of the environment object that is passed to your `createPlugin`
function, there is a `database` field. You can use that to get a
[Knex](http://knexjs.org/) connection object.
```ts
// in packages/backend/src/plugins/carmen.ts
export default async function createPlugin(env: PluginEnvironment) {
const db: Knex<any, unknown[]> = await env.database.getClient();
// You will then pass this client into your actual plugin implementation
// code, maybe similar to the following:
const model = new CarmenDatabaseModel(db);
return await createRouter({
model: model,
logger: env.logger,
});
}
```
You may note that the `getClient` call has no parameters. This is because all
plugin database needs are configured under the `backend.database` config key of
your `app-config.yaml`. The framework may even make sure behind the scenes that
the logical database is created automatically if it doesn't exist, based on
rules that the Backstage operator decides on.
The framework does not handle database schema migrations for you, however. The
builtin plugins in the main repo have chosen to use the Knex library to manage
schema migrations as well, but you can do so in any manner that you see fit.
See the [Knex library documentation](http://knexjs.org/) for examples and
details on how to write schema migrations and perform SQL queries against your
database..
docs/plugins/composability.md
+5 -4
View File
@@ -511,10 +511,11 @@ clarify intent. Refer to the following table to formulate the new name:
## Porting Existing Apps
The first step of porting any app is to replace the root `Routes` component with
`FlatRoutes` from `@backstage/core`. As opposed to the `Routes` component,
`FlatRoutes` only considers the first level of `Route` components in its
children, and provides any additional children to the outlet of the route. It
also removes the need to append `"/*"` to paths, as it is added automatically.
`FlatRoutes` from `@backstage/core-app-api`. As opposed to the `Routes`
component, `FlatRoutes` only considers the first level of `Route` components in
its children, and provides any additional children to the outlet of the route.
It also removes the need to append `"/*"` to paths, as it is added
automatically.
```diff
const AppRoutes = () => (
docs/plugins/create-a-plugin.md
+4 -4
View File
@@ -22,9 +22,9 @@ yarn create-plugin
This will create a new Backstage Plugin based on the ID that was provided. It
will be built and added to the Backstage App automatically.
> If `yarn start` is already running you should be able to see the default page
> for your new plugin directly by navigating to
> `http://localhost:3000/my-plugin`.
> If the Backstage App is already running (with `yarn start` or `yarn dev`) you
> should be able to see the default page for your new plugin directly by
> navigating to `http://localhost:3000/my-plugin`.
![](../assets/my-plugin_screenshot.png)
@@ -32,7 +32,7 @@ You can also serve the plugin in isolation by running `yarn start` in the plugin
directory. Or by using the yarn workspace command, for example:
```bash
yarn workspace @backstage/plugin-welcome start # Also supports --check
yarn workspace @backstage/my-plugin start # Also supports --check
```
This method of serving the plugin provides quicker iteration speed and a faster
docs/plugins/github-apps.md
+13
View File
@@ -43,6 +43,10 @@ root of the project which you can then use as an `include` in your
`app-config.yaml`. You can go ahead and
[skip ahead](#including-in-integrations-config) if you've already got an app.
Note that the created app will have a webhook that is disabled by default and
points to `smee.io`, which is intended for local development. There's also
currently no part of Backstage that makes use of the webhook.
### GitHub Enterprise
You have to create the GitHub Application manually using these
@@ -104,3 +108,12 @@ privateKey: |
This will result in backstage preventing the use of any installation that is not
within the allow list.
### Permissions for pull requests
These are the minimum permissions required for creating a pull request with
Backstage software templates:
- Read and Write permissions for `Contents`.
- Read and write permissions for `Pull Requests` and `Issues`.
- Read permissions on `Metadata`.
docs/plugins/index.md
+4 -4
View File
@@ -28,9 +28,9 @@ This helps the community know what plugins are in development.
You can also use this process if you have an idea for a good plugin but you hope
that someone else will pick up the work.
## Integrate into the Service Catalog
## Integrate into the Software Catalog
If your plugin isn't supposed to live as a standalone page, but rather needs to
be presented as a part of a Service Catalog (e.g. a separate tab or a card on an
"Overview" tab), then check out
[the instruction](integrating-plugin-into-service-catalog.md) on how to do it.
be presented as a part of a Software Catalog (e.g. a separate tab or a card on
an "Overview" tab), then check out
[the instruction](integrating-plugin-into-software-catalog.md) on how to do it.
docs/plugins/integrating-plugin-into-service-catalog.md → docs/plugins/integrating-plugin-into-software-catalog.md
+3 -3
View File
@@ -1,7 +1,7 @@
---
id: integrating-plugin-into-service-catalog
title: Integrate into the Service Catalog
description: How to integrate a plugin into service catalog
id: integrating-plugin-into-software-catalog
title: Integrate into the Software Catalog
description: How to integrate a plugin into software catalog
---
> This is an advanced use case and currently is an experimental feature. Expect
docs/plugins/plugin-development.md
+3 -3
View File
@@ -36,7 +36,7 @@ to avoid import cycles, for example like this:
```tsx
/* src/routes.ts */
import { createRouteRef } from '@backstage/core';
import { createRouteRef } from '@backstage/core-plugin-api';
// Note: This route ref is for internal use only, don't export it from the plugin
export const rootRouteRef = createRouteRef({
@@ -46,11 +46,11 @@ export const rootRouteRef = createRouteRef({
Now that we have a `RouteRef`, we import it into `src/plugin.ts`, create our
plugin instance with `createPlugin`, as well as create and wrap our routable
extension using `createRoutableExtension` from `@backstage/core`:
extension using `createRoutableExtension` from `@backstage/core-plugin-api`:
```tsx
/* src/plugin.ts */
import { createPlugin, createRouteRef } from '@backstage/core';
import { createPlugin, createRouteRef } from '@backstage/core-plugin-api';
import ExampleComponent from './components/ExampleComponent';
// Create a plugin instance and export this from your plugin package
docs/plugins/structure-of-a-plugin.md
+4 -1
View File
@@ -57,7 +57,10 @@ package.json to declare the plugin dependencies, metadata and scripts.
In the `src` folder we get to the interesting bits. Check out the `plugin.ts`:
```jsx
import { createPlugin, createRoutableExtension } from '@backstage/core';
import {
createPlugin,
createRoutableExtension,
} from '@backstage/core-plugin-api';
import { rootRouteRef } from './routes';
docs/plugins/url-reader.md
+274
View File
@@ -0,0 +1,274 @@
---
id: url-reader
title: URL Reader
sidebar_label: URL Reader
# prettier-ignore
description: URL Reader is a backend core API responsible for reading files from external locations.
---
## Concept
Some of the core plugins of Backstage have to read files from an external
location. [Software Catalog](../features/software-catalog/index.md) has to read
the [`catalog-info.yaml`](../features/software-catalog/descriptor-format.md)
entity descriptor files to register and track an entity.
[Software Templates](../features/software-templates/index.md) have to download
the template skeleton files before creating a new component.
[TechDocs](../features/techdocs/README.md) has to download the markdown source
files before generating a documentation site.
Since, the requirement for reading files is so essential for Backstage plugins,
the
[`@backstage/backend-common`](https://github.com/backstage/backstage/tree/master/packages/backend-common)
package provides a dedicated API for reading from such URL based remote
locations like GitHub, GitLab, Bitbucket, Google Cloud Storage, etc. This is
commonly referred to as "URL Reader". It takes care of making authenticated
requests to the remote host so that private files can be read securely. If users
have [GitHub App based authentication](github-apps.md) set up, URL Reader even
refreshes the token, to avoid reaching the GitHub API rate limit.
As a result, plugin authors do not have to worry about any of these problems
when trying to read files.
## Interface
When the Backstage backend starts, a new instance of URL Reader is created. You
can see this in the index file of your Backstage backend
i.e.`packages/backend/src/index.ts`.
[Example](https://github.com/backstage/backstage/blob/ebbe91dbe79038a61d35cf6ed2d96e0e0d5a15f3/packages/backend/src/index.ts#L57)
```ts
// File: packages/backend/src/index.ts
import { URLReaders } from '@backstage/backend-common';
function makeCreateEnv(config: Config) {
// ....
const reader = UrlReaders.default({ logger, config });
//
}
```
This instance contains all
[the default URL Reader providers](https://github.com/backstage/backstage/blob/master/packages/backend-common/src/reading/UrlReaders.ts)
in the backend-common package including GitHub, GitLab, Bitbucket, Azure, Google
GCS. As the need arises, more URL Readers are being written to support different
providers.
The generic interface of a URL Reader instance looks like this.
```ts
export type UrlReader = {
/* Used to read a single file and return its content. */
read(url: string): Promise<Buffer>;
/**
* A replacement for the read method that supports options and complex responses.
*
* Use this whenever it is available, as the read method will be deprecated and
* eventually removed in the future.
*/
readUrl?(url: string, options?: ReadUrlOptions): Promise<ReadUrlResponse>;
/* Used to read a file tree and download as a directory. */
readTree(url: string, options?: ReadTreeOptions): Promise<ReadTreeResponse>;
/* Used to search a file in a tree. */
search(url: string, options?: SearchOptions): Promise<SearchResponse>;
};
```
## Using a URL Reader inside a plugin
The `reader` instance is available in the backend plugin environment and passed
on to all the backend plugins. You can see an
[example](https://github.com/backstage/backstage/blob/b0be185369ebaad22255b7cdf18535d1d4ffd0e7/packages/backend/src/plugins/techdocs.ts#L31).
When any of the methods on this instance is called with a URL, URL Reader
extracts the host for that URL (e.g. `github.com`, `ghe.mycompany.com`, etc.).
Using the
[`@backstage/integration`](https://github.com/backstage/backstage/tree/master/packages/integration)
package, it looks inside the
[`integrations:`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/app-config.yaml#L134-L158)
config of the `app-config.yaml` to find out how to work with the host based on
the configs provided like authentication token, API base URL, etc.
Make sure your plugin-specific backend file at
`packages/backend/src/plugins/<PLUGIN>.ts` is forwarding the `reader` instance
passed on as the `PluginEnvironment` to the actual plugin's `createRouter`
function. See how this is done in
[Catalog](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend/src/plugins/catalog.ts#L25-L27)
and
[TechDocs](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend/src/plugins/techdocs.ts#L31-L36)
backend plugins.
Once the reader instance is available inside the plugin, one of its methods can
directly be used with a URL. Some example usages -
- [`read`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/plugins/catalog-backend/src/ingestion/processors/codeowners/read.ts#L24-L33) -
Catalog using the `read` method to read the CODEOWNERS file in a repository.
- [`readTree`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/techdocs-common/src/helpers.ts#L198-L220) -
TechDocs using the `readTree` method to download markdown files in order to
generate the documentation site.
- [`readTree`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/techdocs-common/src/stages/prepare/url.ts#L33-L54) -
TechDocs using `NotModifiedError` to maintain cache and speed up and limit the
number of requests.
- [`search`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/plugins/catalog-backend/src/ingestion/processors/UrlReaderProcessor.ts#L88-L108) -
Catalog using the `search` method to find files for a location URL containing
a glob pattern.
## Writing a new URL Reader
If the available URL Readers are not sufficient for your use case and you want
to add a new URL Reader for any other provider, you are most welcome to
contribute one!
Feel free to use the
[GitHub URL Reader](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/GithubUrlReader.ts)
as a source of inspiration.
### 1. Add an integration
The provider for your new URL Reader can also be called an "integration" in
Backstage. The `integrations:` section of your Backstage `app-config.yaml`
config file is supposed to be the place where a Backstage integrator defines the
host URL for the integration, authentication details and other integration
related configurations.
The `@backstage/integration` package is where most of the integration specific
code lives, so that it is shareable across Backstage. Functions like "read the
integrations config and process it", "construct headers for authenticated
requests to the host" or "convert a plain file URL into its API URL for
downloading the file" would live in this package.
### 2. Create the URL Reader
Create a new class which implements the
[`UrlReader` type](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/types.ts#L21-L28)
inside `@backstage/backend-common`. Create and export a static `factory` method
which reads the integration config and returns a map of host URLs the new reader
should be used for. See the
[GitHub URL Reader](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/GithubUrlReader.ts#L50-L63)
for example.
### 3. Implement the methods
We want to make sure all URL Readers behave in the same way. Hence if possible,
all the methods of the `UrlReader` interface should be implemented. However it
is okay to start by implementing just one of them and create issues for the
remaining.
#### read
NOTE: Use `readUrl` instead of `read`.
`read` method expects a user-friendly URL, something which can be copied from
the browser naturally when a person is browsing the provider in their browser.
- ✅ Valid URL :
`https://github.com/backstage/backstage/blob/master/ADOPTERS.md`
- ❌ Not a valid URL :
`https://raw.githubusercontent.com/backstage/backstage/master/ADOPTERS.md`
- ❌ Not a valid URL : `https://github.com/backstage/backstage/ADOPTERS.md`
Upon receiving the URL, `read` converts the user-friendly URL into an API URL
which can be used to request the provider's API.
`read` then makes an authenticated request to the provider API and returns the
file's content.
#### readUrl
`readUrl` is a new interface that allows complex response objects and is
intended to replace the `read` method. This new method is currently optional to
implement which allows for a soft migration to `readUrl` instead of `read` in
the future.
#### readTree
`readTree` method also expects user-friendly URLs similar to `read` but the URL
should point to a tree (could be the root of a repository or even a
sub-directory).
- ✅ Valid URL : `https://github.com/backstage/backstage`
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master`
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/docs`
Using the provider's API documentation, find out an API endpoint which can be
used to download either a zip or a tarball. You can download the entire tree
(e.g. a repository) and filter out in case the user is expecting only a
sub-tree. But some APIs are smart enough to accept a path and return only a
sub-tree in the downloaded archive.
#### search
`search` method expects a glob pattern of a URL and returns a list of files
matching the query.
- ✅ Valid URL :
`https://github.com/backstage/backstage/blob/master/**/catalog-info.yaml`
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/**/*.md`
- ✅ Valid URL :
`https://github.com/backstage/backstage/blob/master/*/package.json`
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/READM`
The core logic of `readTree` can be used here to extract all the files inside
the tree and return the files matching the pattern in the `url`.
### 4. Add to available URL Readers
There are two ways to make your new URL Reader available for use.
You can choose to make it open source, by updating the
[`default` factory](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/UrlReaders.ts#L62-L81)
method of URL Readers.
But for something internal which you don't want to make open source, you can
update your `packages/backend/src/index.ts` file and update how the `reader`
instance is created.
```ts
// File: packages/backend/src/index.ts
const reader = UrlReaders.default({
logger: root,
config,
// This is where your internal URL Readers would go.
factories: [myCustomReader.factory],
});
```
### 5. Caching
All of the methods above support an ETag based caching. If the method is called
without an `etag`, the response contains an ETag of the resource (should ideally
forward the ETag returned by the provider). If the method is called with an
`etag`, it first compares the ETag and returns a `NotModifiedError` in case the
resource has not been modified. This approach is very similar to the actual
[ETag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag) and
[If-None-Match](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match)
HTTP headers.
### 6. Debugging
When debugging one of the URL Readers, you can straightforward use the
[`reader` instance created](https://github.com/backstage/backstage/blob/ebbe91dbe79038a61d35cf6ed2d96e0e0d5a15f3/packages/backend/src/index.ts#L57)
when the backend starts and call one of the methods with your debugging URL.
```ts
// File: packages/backend/src/index.ts
async function main() {
// ...
const createEnv = makeCreateEnv(config);
const testReader = createEnv('test-url-reader').reader;
const response = await testReader.readUrl(
'https://github.com/backstage/backstage/blob/master/catalog-info.yaml',
);
console.log((await response.buffer()).toString());
// ...
}
```
This will be run every time you restart the backend. Note that after any change
in the URL Reader code, you need to kill the backend and restart, since the
`reader` instance is memoized and does not update on hot module reloading. Also,
there are a lot of unit tests written for the URL Readers, which you can make
use of.