docs: Document URL Reader

Concept Interface Using inside a plugin Writing a new one Signed-off-by: Himanshu Mishra <himanshu@orkohunter.net>
2021-07-01 22:07:15 +02:00
parent 82b6d84cf3
commit 4d2528ad4a
3 changed files with 222 additions and 1 deletions
@@ -0,0 +1,219 @@
+---
+id: url-reader
+title: URL Reader
+sidebar_label: URL Reader
+# prettier-ignore
+description: URL Reader is a backend core API responsible for reading files from external locations.
+---
+
+## Concept
+
+Some of the core plugins of Backstage have to read files from an external
+location. [Software Catalog](../features/software-catalog/index.md) has to read
+the [`catalog-info.yaml`](../features/software-catalog/descriptor-format.md)
+entity descriptor files to register and track an entity.
+[Software Templates](../features/software-templates/index.md) have to download
+the template skeleton files before creating a new component.
+[TechDocs](../features/techdocs/README.md) has to download the markdown source
+files before generating a documentation site.
+
+Since, the requirement for reading files is so essential for Backstage plugins,
+the
+[`@backstage/backend-common`](https://github.com/backstage/backstage/tree/master/packages/backend-common)
+package provides a dedicated API for reading from such URL based remote
+locations like GitHub, GitLab, Bitbucket, Google Cloud Storage, etc. This is
+commonly referred to as "URL Reader". It takes care of making authenticated
+requests to the remote host so that private files can be read securely. If users
+have [GitHub App based authentication](../plugins/github-apps.md) set up, URL
+Reader even refreshes the token, to avoid reaching the GitHub API rate limit.
+
+As a result, plugin authors do not have to worry about any of these problems
+when trying to read files.
+
+## Interface
+
+When the Backstage backend starts, a new instance of URL Reader is created. You
+can see this in the index file of your Backstage backend
+i.e.`packages/backend/src/index.ts`.
+[Example](https://github.com/backstage/backstage/blob/ebbe91dbe79038a61d35cf6ed2d96e0e0d5a15f3/packages/backend/src/index.ts#L57)
+
+```ts
+// File: packages/backend/src/index.ts
+
+import { URLReaders } from '@backstage/backend-common';
+
+function makeCreateEnv(config: Config) {
+  // ....
+  const reader = UrlReaders.default({ logger, config });
+  //
+}
+```
+
+This instance contains all
+[the default URL Reader providers](https://github.com/backstage/backstage/blob/master/packages/backend-common/src/reading/UrlReaders.ts)
+in the backend-common package including GitHub, Gitlab, Bitbucket, Azure, Google
+GCS. As the need arises, more URL Readers are being written to support different
+providers.
+
+The generic interface of a URL Reader instance looks like this.
+
+```ts
+export type UrlReader = {
+  /* Used to read a single file and return its content. */
+  read(url: string): Promise<Buffer>;
+  /* Used to read a file tree and download as a directory. */
+  readTree(url: string, options?: ReadTreeOptions): Promise<ReadTreeResponse>;
+  /* Used to search a file in a tree. */
+  search(url: string, options?: SearchOptions): Promise<SearchResponse>;
+};
+```
+
+## Using inside a plugin
+
+The `reader` instance is available in the backend plugin environment and passed
+on to all the backend plugins. You can see an
+[example](https://github.com/backstage/backstage/blob/b0be185369ebaad22255b7cdf18535d1d4ffd0e7/packages/backend/src/plugins/techdocs.ts#L31).
+When any of the methods on this instance is called with a URL, URL Reader
+extracts the host for that URL (e.g. `github.com`, `ghe.mycompany.com`, etc.).
+Using the
+[`@backstage/integration`](https://github.com/backstage/backstage/tree/master/packages/integration)
+package, it looks inside the
+[`integrations:`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/app-config.yaml#L134-L158)
+config of the `app-config.yaml` to find out how to work with the host based on
+the configs provided like authentication token, API base URL, etc.
+
+Make sure your plugin-specific backend file at
+`packages/backend/src/plugins/<PLUGIN>.ts` is forwarding the `reader` instance
+passed on as the `PluginEnvironment` to the actual plugin's `createRouter`
+function. See how this is done in
+[Catalog](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend/src/plugins/catalog.ts#L25-L27)
+and
+[TechDocs](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend/src/plugins/techdocs.ts#L31-L36)
+backend plugins.
+
+Once the reader instance is available inside the plugin, one of its methods can
+directly be used with a URL. Some example usages -
+
+- [`read`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/plugins/catalog-backend/src/ingestion/processors/codeowners/read.ts#L24-L33) -
+  Catalog using the `read` method to read the CODEOWNERS file in a repository.
+- [`readTree`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/techdocs-common/src/helpers.ts#L198-L220) -
+  TechDocs using the `readTree` method to download markdown files in order to
+  generate the documentation site.
+- [`readTree`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/techdocs-common/src/stages/prepare/url.ts#L33-L54) -
+  TechDocs using `NotModifiedError` to maintain cache and speed up and limit the
+  number of requests.
+- [`search`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/plugins/catalog-backend/src/ingestion/processors/UrlReaderProcessor.ts#L88-L108) -
+  Catalog using the `search` method to find files for a location URL containing
+  a glob pattern.
+
+## Writing a new URL Reader
+
+If the available URL Readers are not sufficient for your use case and you want
+to add a new URL Reader for any other provider, you are most welcome to
+contribute one!
+
+Feel free to use the
+[GitHub URL Reader](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/GithubUrlReader.ts)
+as a source of inspiration.
+
+### Add an integration
+
+The provider for your new URL Reader can also be called an "integration" in
+Backstage. The `integrations:` section of your Backstage `app-config.yaml`
+config file is supposed to be the place where a Backstage integrator defines the
+host URL for the integration, authentication details and other integration
+related configurations.
+
+The `@backstage/integration` package is where most of the integration specific
+code lives, so that it is shareable across Backstage. Functions like "read the
+integrations config and process it", "construct headers for authenticated
+requests to the host" or "convert a plain file URL into its API URL for
+downloading the file" would live in this package.
+
+### Create the URL Reader
+
+Create a new class which implements the
+[`UrlReader` type](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/types.ts#L21-L28)
+inside `@backstage/backend-common`. Create and export a static `factory` method
+which reads the integration config and returns a map of host URLs the new reader
+should be used for. See the
+[GitHub URL Reader](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/packages/backend-common/src/reading/GithubUrlReader.ts#L50-L63)
+for example.
+
+### Implement the methods
+
+We want to make sure all URL Readers behave in the same way. Hence if possible,
+all the methods of the `UrlReader` interface should be implemented. However it
+is okay to start by implementing just one of them and create issues for the
+remaining.
+
+#### read
+
+`read` method expects a user-friendly URL, something which can be copied from
+the browser naturally when a person is browsing the provider in their browser.
+
+- ✅ Valid URL :
+  `https://github.com/backstage/backstage/blob/master/ADOPTERS.md`
+- ❌ Not a valid URL :
+  `https://raw.githubusercontent.com/backstage/backstage/master/ADOPTERS.md`
+- ❌ Not a valid URL : `https://github.com/backstage/backstage/ADOPTERS.md`
+
+Upon receiving the URL, `read` converts the user-friendly URL into an API URL
+which can be used to request the provider's API.
+
+`read` then makes an authenticated request to the provider API and returns the
+file's content.
+
+#### readTree
+
+`readTree` method also expects user-friendly URLs similar to `read` but the URL
+should point to a tree (could be the root of a repository or even a
+sub-directory).
+
+- ✅ Valid URL : `https://github.com/backstage/backstage`
+- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master`
+- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/docs`
+
+Using the provider's API documentation, find out an API endpoint which can be
+used to download either a zip or a tarball. You can download the entire tree
+(e.g. a repository) and filter out in case the user is expecting only a
+sub-tree. But some APIs are smart enough to accept a path and return only a
+sub-tree in the downloaded archive.
+
+#### search
+
+`search` method expects a glob pattern of a URL and returns a list of files
+matching the query.
+
+- ✅ Valid URL :
+  `https://github.com/backstage/backstage/blob/master/**/catalog-info.yaml`
+- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/**/*.md`
+- ✅ Valid URL :
+  `https://github.com/backstage/backstage/blob/master/*/package.json`
+- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/READM`
+
+The core logic of `readTree` can be used here to extract all the files inside
+the tree and return the files matching the pattern in the `url`.
+
+### Caching
+
+All of the methods above support an ETag based caching (except `read` which is
+work in progress!). If the method is called without an `etag`, the response
+contains an ETag of the resource (should ideally forward the ETag returned by
+the provider). If the method is called with an `etag`, it first compares the
+ETag and returns a `NotModifiedError` in case the resource has not been
+modified. This approach is very similar to the actual
+[ETag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag) and
+[If-None-Match](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match)
+HTTP headers.
+
+### Debugging
+
+When debugging one of the URL Readers, you can straightforward use the
+[`reader` instance created](https://github.com/backstage/backstage/blob/ebbe91dbe79038a61d35cf6ed2d96e0e0d5a15f3/packages/backend/src/index.ts#L57)
+when the backend starts and call one of the methods with your debugging URL.
+This will be run every time you restart the backend. Note that after any change
+in the URL Reader code, you need to kill the backend and restart, since the
+`reader` instance is memoized and does not update on hot module reloading. Also,
+there are a lot of unit tests written for the URL Readers, which you can make
+use of.
@@ -149,7 +149,8 @@
        "type": "subcategory",
        "label": "LDAP",
        "ids": ["integrations/ldap/org"]
-      }
+      },
+      "integrations/url-reader"
    ],
    "Plugins": [
      "plugins/index",
@@ -98,6 +98,7 @@ nav:
          - Locations: 'integrations/google-cloud-storage/locations.md'
      - LDAP:
          - Org Data: 'integrations/ldap/org.md'
+      - URL Reader: 'integrations/url-reader.md'
  - Plugins:
      - Intro to plugins: 'plugins/index.md'
      - Existing plugins: 'plugins/existing-plugins.md'