From 4d2528ad4aced5c8163880ce7aa94e3e89d96238 Mon Sep 17 00:00:00 2001 From: Himanshu Mishra Date: Thu, 1 Jul 2021 22:07:15 +0200 Subject: [PATCH] docs: Document URL Reader Concept Interface Using inside a plugin Writing a new one Signed-off-by: Himanshu Mishra --- docs/integrations/url-reader.md | 219 ++++++++++++++++++++++++++++++++ microsite/sidebars.json | 3 +- mkdocs.yml | 1 + 3 files changed, 222 insertions(+), 1 deletion(-) create mode 100644 docs/integrations/url-reader.md diff --git a/docs/integrations/url-reader.md b/docs/integrations/url-reader.md new file mode 100644 index 0000000000..e9d6caf17c --- /dev/null +++ b/docs/integrations/url-reader.md @@ -0,0 +1,219 @@ +--- +id: url-reader +title: URL Reader +sidebar_label: URL Reader +# prettier-ignore +description: URL Reader is a backend core API responsible for reading files from external locations. +--- + +## Concept + +Some of the core plugins of Backstage have to read files from an external +location. [Software Catalog](../features/software-catalog/index.md) has to read +the [`catalog-info.yaml`](../features/software-catalog/descriptor-format.md) +entity descriptor files to register and track an entity. +[Software Templates](../features/software-templates/index.md) have to download +the template skeleton files before creating a new component. +[TechDocs](../features/techdocs/README.md) has to download the markdown source +files before generating a documentation site. + +Since, the requirement for reading files is so essential for Backstage plugins, +the +[`@backstage/backend-common`](https://github.com/backstage/backstage/tree/master/packages/backend-common) +package provides a dedicated API for reading from such URL based remote +locations like GitHub, GitLab, Bitbucket, Google Cloud Storage, etc. This is +commonly referred to as "URL Reader". It takes care of making authenticated +requests to the remote host so that private files can be read securely. If users +have [GitHub App based authentication](../plugins/github-apps.md) set up, URL +Reader even refreshes the token, to avoid reaching the GitHub API rate limit. + +As a result, plugin authors do not have to worry about any of these problems +when trying to read files. + +## Interface + +When the Backstage backend starts, a new instance of URL Reader is created. You +can see this in the index file of your Backstage backend +i.e.`packages/backend/src/index.ts`. +[Example](https://github.com/backstage/backstage/blob/ebbe91dbe79038a61d35cf6ed2d96e0e0d5a15f3/packages/backend/src/index.ts#L57) + +```ts +// File: packages/backend/src/index.ts + +import { URLReaders } from '@backstage/backend-common'; + +function makeCreateEnv(config: Config) { + // .... + const reader = UrlReaders.default({ logger, config }); + // +} +``` + +This instance contains all +[the default URL Reader providers](https://github.com/backstage/backstage/blob/master/packages/backend-common/src/reading/UrlReaders.ts) +in the backend-common package including GitHub, Gitlab, Bitbucket, Azure, Google +GCS. As the need arises, more URL Readers are being written to support different +providers. + +The generic interface of a URL Reader instance looks like this. + +```ts +export type UrlReader = { + /* Used to read a single file and return its content. */ + read(url: string): Promise; + /* Used to read a file tree and download as a directory. */ + readTree(url: string, options?: ReadTreeOptions): Promise; + /* Used to search a file in a tree. */ + search(url: string, options?: SearchOptions): Promise; +}; +``` + +## Using inside a plugin + +The `reader` instance is available in the backend plugin environment and passed +on to all the backend plugins. You can see an +[example](https://github.com/backstage/backstage/blob/b0be185369ebaad22255b7cdf18535d1d4ffd0e7/packages/backend/src/plugins/techdocs.ts#L31). +When any of the methods on this instance is called with a URL, URL Reader +extracts the host for that URL (e.g. `github.com`, `ghe.mycompany.com`, etc.). +Using the +[`@backstage/integration`](https://github.com/backstage/backstage/tree/master/packages/integration) +package, it looks inside the +[`integrations:`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/app-config.yaml#L134-L158) +config of the `app-config.yaml` to find out how to work with the host based on +the configs provided like authentication token, API base URL, etc. + +Make sure your plugin-specific backend file at +`packages/backend/src/plugins/.ts` is forwarding the `reader` instance +passed on as the `PluginEnvironment` to the actual plugin's `createRouter` +function. See how this is done in +[Catalog](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend/src/plugins/catalog.ts#L25-L27) +and +[TechDocs](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend/src/plugins/techdocs.ts#L31-L36) +backend plugins. + +Once the reader instance is available inside the plugin, one of its methods can +directly be used with a URL. Some example usages - + +- [`read`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/plugins/catalog-backend/src/ingestion/processors/codeowners/read.ts#L24-L33) - + Catalog using the `read` method to read the CODEOWNERS file in a repository. +- [`readTree`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/techdocs-common/src/helpers.ts#L198-L220) - + TechDocs using the `readTree` method to download markdown files in order to + generate the documentation site. +- [`readTree`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/techdocs-common/src/stages/prepare/url.ts#L33-L54) - + TechDocs using `NotModifiedError` to maintain cache and speed up and limit the + number of requests. +- [`search`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/plugins/catalog-backend/src/ingestion/processors/UrlReaderProcessor.ts#L88-L108) - + Catalog using the `search` method to find files for a location URL containing + a glob pattern. + +## Writing a new URL Reader + +If the available URL Readers are not sufficient for your use case and you want +to add a new URL Reader for any other provider, you are most welcome to +contribute one! + +Feel free to use the +[GitHub URL Reader](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/GithubUrlReader.ts) +as a source of inspiration. + +### Add an integration + +The provider for your new URL Reader can also be called an "integration" in +Backstage. The `integrations:` section of your Backstage `app-config.yaml` +config file is supposed to be the place where a Backstage integrator defines the +host URL for the integration, authentication details and other integration +related configurations. + +The `@backstage/integration` package is where most of the integration specific +code lives, so that it is shareable across Backstage. Functions like "read the +integrations config and process it", "construct headers for authenticated +requests to the host" or "convert a plain file URL into its API URL for +downloading the file" would live in this package. + +### Create the URL Reader + +Create a new class which implements the +[`UrlReader` type](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/types.ts#L21-L28) +inside `@backstage/backend-common`. Create and export a static `factory` method +which reads the integration config and returns a map of host URLs the new reader +should be used for. See the +[GitHub URL Reader](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/GithubUrlReader.ts#L50-L63) +for example. + +### Implement the methods + +We want to make sure all URL Readers behave in the same way. Hence if possible, +all the methods of the `UrlReader` interface should be implemented. However it +is okay to start by implementing just one of them and create issues for the +remaining. + +#### read + +`read` method expects a user-friendly URL, something which can be copied from +the browser naturally when a person is browsing the provider in their browser. + +- ✅ Valid URL : + `https://github.com/backstage/backstage/blob/master/ADOPTERS.md` +- ❌ Not a valid URL : + `https://raw.githubusercontent.com/backstage/backstage/master/ADOPTERS.md` +- ❌ Not a valid URL : `https://github.com/backstage/backstage/ADOPTERS.md` + +Upon receiving the URL, `read` converts the user-friendly URL into an API URL +which can be used to request the provider's API. + +`read` then makes an authenticated request to the provider API and returns the +file's content. + +#### readTree + +`readTree` method also expects user-friendly URLs similar to `read` but the URL +should point to a tree (could be the root of a repository or even a +sub-directory). + +- ✅ Valid URL : `https://github.com/backstage/backstage` +- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master` +- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/docs` + +Using the provider's API documentation, find out an API endpoint which can be +used to download either a zip or a tarball. You can download the entire tree +(e.g. a repository) and filter out in case the user is expecting only a +sub-tree. But some APIs are smart enough to accept a path and return only a +sub-tree in the downloaded archive. + +#### search + +`search` method expects a glob pattern of a URL and returns a list of files +matching the query. + +- ✅ Valid URL : + `https://github.com/backstage/backstage/blob/master/**/catalog-info.yaml` +- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/**/*.md` +- ✅ Valid URL : + `https://github.com/backstage/backstage/blob/master/*/package.json` +- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/READM` + +The core logic of `readTree` can be used here to extract all the files inside +the tree and return the files matching the pattern in the `url`. + +### Caching + +All of the methods above support an ETag based caching (except `read` which is +work in progress!). If the method is called without an `etag`, the response +contains an ETag of the resource (should ideally forward the ETag returned by +the provider). If the method is called with an `etag`, it first compares the +ETag and returns a `NotModifiedError` in case the resource has not been +modified. This approach is very similar to the actual +[ETag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag) and +[If-None-Match](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match) +HTTP headers. + +### Debugging + +When debugging one of the URL Readers, you can straightforward use the +[`reader` instance created](https://github.com/backstage/backstage/blob/ebbe91dbe79038a61d35cf6ed2d96e0e0d5a15f3/packages/backend/src/index.ts#L57) +when the backend starts and call one of the methods with your debugging URL. +This will be run every time you restart the backend. Note that after any change +in the URL Reader code, you need to kill the backend and restart, since the +`reader` instance is memoized and does not update on hot module reloading. Also, +there are a lot of unit tests written for the URL Readers, which you can make +use of. diff --git a/microsite/sidebars.json b/microsite/sidebars.json index 0231d8f692..5530fc6f61 100644 --- a/microsite/sidebars.json +++ b/microsite/sidebars.json @@ -149,7 +149,8 @@ "type": "subcategory", "label": "LDAP", "ids": ["integrations/ldap/org"] - } + }, + "integrations/url-reader" ], "Plugins": [ "plugins/index", diff --git a/mkdocs.yml b/mkdocs.yml index c3d77e179f..47b7084146 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -98,6 +98,7 @@ nav: - Locations: 'integrations/google-cloud-storage/locations.md' - LDAP: - Org Data: 'integrations/ldap/org.md' + - URL Reader: 'integrations/url-reader.md' - Plugins: - Intro to plugins: 'plugins/index.md' - Existing plugins: 'plugins/existing-plugins.md'