Consolidate url reader docs
Signed-off-by: Johan Haals <johan.haals@gmail.com>
This commit is contained in:
Johan Haals
@@ -45,3 +45,99 @@ createBackendPlugin({
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Providing custom URL readers
|
||||
|
||||
Custom URL readers that are not intended for open sourcing via a custom service factory for that specific reader. The following example shows how to create a custom URL reader and provide it to the backend.
|
||||
|
||||
```ts
|
||||
// File: packages/backend/src/index.ts
|
||||
import { urlReaderFactoriesServiceRef } from '@backstage/backend-defaults/urlReader';
|
||||
import {
|
||||
createBackendFeatureLoader,
|
||||
createServiceFactory,
|
||||
} from '@backstage/backend-plugin-api';
|
||||
|
||||
const customReader = createServiceFactory({
|
||||
service: urlReaderFactoriesServiceRef,
|
||||
deps: {},
|
||||
async factory() {
|
||||
return CustomUrlReader.factory;
|
||||
},
|
||||
});
|
||||
|
||||
const backend = createBackend();
|
||||
// backend.add() of other plugins and modules excluded
|
||||
backend.add(customReader);
|
||||
```
|
||||
|
||||
## Writing URL Readers
|
||||
|
||||
We want to make sure all URL Readers behave in the same way. Hence if possible,
|
||||
all the methods of the `UrlReaderService` interface should be implemented. However it
|
||||
is okay to start by implementing just one of them and create issues for the
|
||||
remaining.
|
||||
|
||||
You can choose to make new URL Readers open source if the use case is beneficial to other users. Either as its own package or by updating the
|
||||
[`default` factory](https://github.com/backstage/backstage/blob/ce2ca68f07ad3334401d3277b989bf145b728a64/packages/backend-defaults/src/entrypoints/urlReader/lib/UrlReaders.ts#L82-L102)
|
||||
method of URL Readers. It's recommended to create an issue in the Backstage repository to discuss the use case and get feedback before starting the implementation of a new core URL Reader.
|
||||
|
||||
Here are some general guidelines for writing URL Readers
|
||||
|
||||
#### `readUrl`
|
||||
|
||||
`readUrl` method expects a user-friendly URL, something which can be copied from
|
||||
the browser naturally when a person is browsing the provider in their browser.
|
||||
|
||||
- ✅ Valid URL :
|
||||
`https://github.com/backstage/backstage/blob/master/ADOPTERS.md`
|
||||
- ❌ Not a valid URL :
|
||||
`https://raw.githubusercontent.com/backstage/backstage/master/ADOPTERS.md`
|
||||
- ❌ Not a valid URL : `https://github.com/backstage/backstage/ADOPTERS.md`
|
||||
|
||||
Upon receiving the URL, `readUrl` converts the user-friendly URL into an API URL
|
||||
which can be used to request the provider's API.
|
||||
|
||||
`readUrl` then makes an authenticated request to the provider API and returns the response containing the file's contents and ETag(if the provider supports it).
|
||||
|
||||
#### `readTree`
|
||||
|
||||
`readTree` method also expects user-friendly URLs similar to `read` but the URL
|
||||
should point to a tree (could be the root of a repository or even a
|
||||
sub-directory).
|
||||
|
||||
- ✅ Valid URL : `https://github.com/backstage/backstage`
|
||||
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master`
|
||||
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/docs`
|
||||
|
||||
Using the provider's API documentation, find out an API endpoint which can be
|
||||
used to download either a zip or a tarball. You can download the entire tree
|
||||
(e.g. a repository) and filter out in case the user is expecting only a
|
||||
sub-tree. But some APIs are smart enough to accept a path and return only a
|
||||
sub-tree in the downloaded archive.
|
||||
|
||||
#### search
|
||||
|
||||
`search` method expects a glob pattern of a URL and returns a list of files
|
||||
matching the query.
|
||||
|
||||
- ✅ Valid URL :
|
||||
`https://github.com/backstage/backstage/blob/master/**/catalog-info.yaml`
|
||||
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/**/*.md`
|
||||
- ✅ Valid URL :
|
||||
`https://github.com/backstage/backstage/blob/master/*/package.json`
|
||||
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/READM`
|
||||
|
||||
The core logic of `readTree` can be used here to extract all the files inside
|
||||
the tree and return the files matching the pattern in the `url`.
|
||||
|
||||
### Caching
|
||||
|
||||
All of the methods above support an ETag based caching. If the method is called
|
||||
without an `etag`, the response contains an ETag of the resource (should ideally
|
||||
forward the ETag returned by the provider). If the method is called with an
|
||||
`etag`, it first compares the ETag and returns a `NotModifiedError` in case the
|
||||
resource has not been modified. This approach is very similar to the actual
|
||||
[`ETag`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag) and
|
||||
[`If-None-Match`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match)
|
||||
HTTP headers.
|
||||
|
||||
@@ -1,306 +0,0 @@
|
||||
---
|
||||
id: url-reader
|
||||
title: URL Reader
|
||||
sidebar_label: URL Reader
|
||||
# prettier-ignore
|
||||
description: URL Reader is a backend core service responsible for reading files from external locations.
|
||||
---
|
||||
|
||||
## Concept
|
||||
|
||||
Some of the core plugins of Backstage have to read files from an external
|
||||
location. [Software Catalog](../features/software-catalog/index.md) has to read
|
||||
the [`catalog-info.yaml`](../features/software-catalog/descriptor-format.md)
|
||||
entity descriptor files to register and track an entity.
|
||||
[Software Templates](../features/software-templates/index.md) have to download
|
||||
the template skeleton files before creating a new component.
|
||||
[TechDocs](../features/techdocs/README.md) has to download the markdown source
|
||||
files before generating a documentation site.
|
||||
|
||||
Since, the requirement for reading files is so essential for Backstage plugins,
|
||||
the
|
||||
[`coreServices.urlReader`](../reference/backend-plugin-api.coreservices.urlreader.md)
|
||||
package provides a dedicated API for reading from such URL based remote
|
||||
locations like GitHub, GitLab, Bitbucket, Google Cloud Storage, etc. This is
|
||||
commonly referred to as "URL Reader". It takes care of making authenticated
|
||||
requests to the remote host so that private files can be read securely. If users
|
||||
have [GitHub App based authentication](../integrations/github/github-apps.md) set up, URL Reader even
|
||||
refreshes the token, to avoid reaching the GitHub API rate limit.
|
||||
|
||||
As a result, plugin authors do not have to worry about any of these problems
|
||||
when trying to read files.
|
||||
|
||||
## Interface
|
||||
|
||||
This service instance contains all the default URL Reader providers
|
||||
in the backend-defaults package including GitHub, GitLab, Bitbucket, Azure, Google
|
||||
GCS. As the need arises, more URL Readers are being written to support different
|
||||
providers.
|
||||
|
||||
The generic interface of a URL Reader instance looks like this.
|
||||
|
||||
```ts
|
||||
export interface UrlReaderService {
|
||||
/**
|
||||
* Reads a single file and return its content.
|
||||
*/
|
||||
readUrl(
|
||||
url: string,
|
||||
options?: UrlReaderServiceReadUrlOptions,
|
||||
): Promise<UrlReaderServiceReadUrlResponse>;
|
||||
|
||||
/**
|
||||
* Reads a full or partial file tree.
|
||||
*/
|
||||
readTree(
|
||||
url: string,
|
||||
options?: UrlReaderServiceReadTreeOptions,
|
||||
): Promise<UrlReaderServiceReadTreeResponse>;
|
||||
|
||||
/**
|
||||
* Searches for a file in a tree using a glob pattern.
|
||||
*/
|
||||
search(
|
||||
url: string,
|
||||
options?: UrlReaderServiceSearchOptions,
|
||||
): Promise<UrlReaderServiceSearchResponse>;
|
||||
}
|
||||
```
|
||||
|
||||
## Using the URL Reader service inside a plugin
|
||||
|
||||
The following example shows how to get the URL Reader service in your `example` backend plugin to read a file and a directory from a GitHub repository.
|
||||
|
||||
```ts
|
||||
import {
|
||||
coreServices,
|
||||
createBackendPlugin,
|
||||
} from '@backstage/backend-plugin-api';
|
||||
import os from 'os';
|
||||
|
||||
createBackendPlugin({
|
||||
pluginId: 'example',
|
||||
register(env) {
|
||||
env.registerInit({
|
||||
deps: {
|
||||
urlReader: coreServices.urlReader,
|
||||
},
|
||||
async init({ urlReader }) {
|
||||
const buffer = await urlReader
|
||||
.read('https://github.com/backstage/backstage/blob/master/README.md')
|
||||
.then(r => r.buffer());
|
||||
|
||||
const tmpDir = os.tmpdir();
|
||||
const directory = await urlReader
|
||||
.readTree(
|
||||
'https://github.com/backstage/backstage/tree/master/packages/backend',
|
||||
)
|
||||
.then(tree => tree.dir({ targetDir: tmpDir }));
|
||||
},
|
||||
});
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
When any of the methods on this instance is called with a URL, URL Reader
|
||||
extracts the host for that URL (e.g. `github.com`, `ghe.mycompany.com`, etc.).
|
||||
Using the
|
||||
[`@backstage/integration`](https://github.com/backstage/backstage/tree/master/packages/integration)
|
||||
package, it looks inside the
|
||||
[`integrations:`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/app-config.yaml#L134-L158)
|
||||
config of the `app-config.yaml` to find out how to work with the host based on
|
||||
the configs provided like authentication token, API base URL, etc.
|
||||
|
||||
Once the reader instance is available inside the plugin, one of its methods can
|
||||
directly be used with a URL. Some example usages -
|
||||
|
||||
- [`readUrl`](https://github.com/backstage/backstage/blob/38f3827/plugins/catalog-backend/src/modules/codeowners/lib/read.ts#L24-L34) -
|
||||
Catalog using the `readUrl` method to read the CODEOWNERS file in a repository.
|
||||
- [`readTree`](https://github.com/backstage/backstage/blob/33ebb28/plugins/techdocs-node/src/helpers.ts#L147-L165) -
|
||||
TechDocs using the `readTree` method to download markdown files in order to
|
||||
generate the documentation site.
|
||||
- [`readTree`](https://github.com/backstage/backstage/blob/33ebb28/plugins/techdocs-node/src/stages/prepare/url.ts#L55-L78) -
|
||||
TechDocs using `NotModifiedError` to maintain cache and speed up and limit the
|
||||
number of requests.
|
||||
- [`search`](https://github.com/backstage/backstage/blob/38f3827/plugins/catalog-backend/src/modules/core/UrlReaderProcessor.ts#L120-L144) -
|
||||
Catalog using the `search` method to find files for a location URL containing
|
||||
a glob pattern.
|
||||
|
||||
Note that URL Readers which target git-based version control systems may, under
|
||||
the hood, leverage the ability to create tar archives based on a specific git
|
||||
commit-ish. A consequence of this is that files and directories configured with
|
||||
[the `export-ignore` attribute](https://git-scm.com/docs/gitattributes#_creating_an_archive)
|
||||
via `.gitattributes` will not be visible to the URL reader when using the
|
||||
`readTree` or `search` methods.
|
||||
|
||||
Be aware of this limitation and ensure that end-users of your plugin are also
|
||||
aware via, for example, documentation.
|
||||
|
||||
## Writing a new URL Reader
|
||||
|
||||
If the available URL Readers are not sufficient for your use case and you want
|
||||
to add a new URL Reader for any other provider, you are most welcome to
|
||||
contribute one!
|
||||
|
||||
Feel free to use the
|
||||
[GitHub URL Reader service](https://github.com/backstage/backstage/blob/ce2ca68f07ad3334401d3277b989bf145b728a64/packages/backend-defaults/src/entrypoints/urlReader/lib/GithubUrlReader.ts#L60)
|
||||
as a source of inspiration.
|
||||
|
||||
### 1. Add an integration
|
||||
|
||||
The provider for your new URL Reader can also be called an "integration" in
|
||||
Backstage. The `integrations:` section of your Backstage `app-config.yaml`
|
||||
config file is supposed to be the place where a Backstage integrator defines the
|
||||
host URL for the integration, authentication details and other integration
|
||||
related configurations.
|
||||
|
||||
The `@backstage/backend-defaults` package is where the URL Reader specific
|
||||
code lives. Functions like "read the integrations config and process it",
|
||||
"construct headers for authenticated requests to the host" or
|
||||
"convert a plain file URL into its API URL for downloading the file"
|
||||
would live in `@backstage/integrations` so that it is sharable across Backstage.
|
||||
|
||||
### 2. Create the URL Reader
|
||||
|
||||
Create a new class which implements the
|
||||
[`UrlReaderService` type](https://github.com/backstage/backstage/blob/c4b8169/packages/backend-plugin-api/src/services/definitions/UrlReaderService.ts#L26)
|
||||
inside `@backstage/backend-defaults`. Create and export a static `factory` method
|
||||
which reads the integration config and returns a map of host URLs the new reader
|
||||
should be used for. See the
|
||||
[GitHub URL Reader](https://github.com/backstage/backstage/blob/ce2ca68f07ad3334401d3277b989bf145b728a64/packages/backend-defaults/src/entrypoints/urlReader/lib/GithubUrlReader.ts#L61-L73)
|
||||
for example.
|
||||
|
||||
### 3. Implement the methods
|
||||
|
||||
We want to make sure all URL Readers behave in the same way. Hence if possible,
|
||||
all the methods of the `UrlReaderService` interface should be implemented. However it
|
||||
is okay to start by implementing just one of them and create issues for the
|
||||
remaining.
|
||||
|
||||
#### `readUrl`
|
||||
|
||||
`readUrl` method expects a user-friendly URL, something which can be copied from
|
||||
the browser naturally when a person is browsing the provider in their browser.
|
||||
|
||||
- ✅ Valid URL :
|
||||
`https://github.com/backstage/backstage/blob/master/ADOPTERS.md`
|
||||
- ❌ Not a valid URL :
|
||||
`https://raw.githubusercontent.com/backstage/backstage/master/ADOPTERS.md`
|
||||
- ❌ Not a valid URL : `https://github.com/backstage/backstage/ADOPTERS.md`
|
||||
|
||||
Upon receiving the URL, `readUrl` converts the user-friendly URL into an API URL
|
||||
which can be used to request the provider's API.
|
||||
|
||||
`readUrl` then makes an authenticated request to the provider API and returns the response containing the file's contents and ETag(if the provider supports it).
|
||||
|
||||
#### `readTree`
|
||||
|
||||
`readTree` method also expects user-friendly URLs similar to `read` but the URL
|
||||
should point to a tree (could be the root of a repository or even a
|
||||
sub-directory).
|
||||
|
||||
- ✅ Valid URL : `https://github.com/backstage/backstage`
|
||||
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master`
|
||||
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/docs`
|
||||
|
||||
Using the provider's API documentation, find out an API endpoint which can be
|
||||
used to download either a zip or a tarball. You can download the entire tree
|
||||
(e.g. a repository) and filter out in case the user is expecting only a
|
||||
sub-tree. But some APIs are smart enough to accept a path and return only a
|
||||
sub-tree in the downloaded archive.
|
||||
|
||||
#### search
|
||||
|
||||
`search` method expects a glob pattern of a URL and returns a list of files
|
||||
matching the query.
|
||||
|
||||
- ✅ Valid URL :
|
||||
`https://github.com/backstage/backstage/blob/master/**/catalog-info.yaml`
|
||||
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/**/*.md`
|
||||
- ✅ Valid URL :
|
||||
`https://github.com/backstage/backstage/blob/master/*/package.json`
|
||||
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/READM`
|
||||
|
||||
The core logic of `readTree` can be used here to extract all the files inside
|
||||
the tree and return the files matching the pattern in the `url`.
|
||||
|
||||
### 4. Add to available URL Readers
|
||||
|
||||
There are two ways to make your new URL Reader available for use.
|
||||
|
||||
You can choose to make it open source, by updating the
|
||||
[`default` factory](https://github.com/backstage/backstage/blob/ce2ca68f07ad3334401d3277b989bf145b728a64/packages/backend-defaults/src/entrypoints/urlReader/lib/UrlReaders.ts#L82-L102)
|
||||
method of URL Readers.
|
||||
|
||||
But for something internal which you don't want to make open source, you can
|
||||
update your `packages/backend/src/index.ts` file and update how the `reader`
|
||||
instance is created.
|
||||
|
||||
```ts
|
||||
// File: packages/backend/src/index.ts
|
||||
import { urlReaderFactoriesServiceRef } from '@backstage/backend-defaults/urlReader';
|
||||
import {
|
||||
createBackendFeatureLoader,
|
||||
createServiceFactory,
|
||||
} from '@backstage/backend-plugin-api';
|
||||
|
||||
const customReader = createServiceFactory({
|
||||
service: urlReaderFactoriesServiceRef,
|
||||
deps: {},
|
||||
async factory() {
|
||||
return CustomUrlReader.factory;
|
||||
},
|
||||
});
|
||||
|
||||
const backend = createBackend();
|
||||
// backend.add() of other plugins and modules excluded
|
||||
backend.add(customReader);
|
||||
```
|
||||
|
||||
### 5. Caching
|
||||
|
||||
All of the methods above support an ETag based caching. If the method is called
|
||||
without an `etag`, the response contains an ETag of the resource (should ideally
|
||||
forward the ETag returned by the provider). If the method is called with an
|
||||
`etag`, it first compares the ETag and returns a `NotModifiedError` in case the
|
||||
resource has not been modified. This approach is very similar to the actual
|
||||
[`ETag`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag) and
|
||||
[`If-None-Match`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match)
|
||||
HTTP headers.
|
||||
|
||||
### 6. Debugging
|
||||
|
||||
When debugging one of the URL Readers, you can straightforward use the
|
||||
[`reader` instance created](https://github.com/backstage/backstage/blob/ebbe91dbe79038a61d35cf6ed2d96e0e0d5a15f3/packages/backend/src/index.ts#L57)
|
||||
when the backend starts and call one of the methods with your debugging URL.
|
||||
|
||||
```ts
|
||||
// File: packages/backend/src/index.ts
|
||||
import {
|
||||
coreServices,
|
||||
createBackendPlugin,
|
||||
} from '@backstage/backend-plugin-api';
|
||||
|
||||
const demoPlugin = createBackendPlugin({
|
||||
pluginId: 'demo',
|
||||
register(env) {
|
||||
env.registerInit({
|
||||
deps: {
|
||||
urlReader: coreServices.urlReader,
|
||||
},
|
||||
async init({ urlReader }) {
|
||||
const resp = urlReader.read('http://my-url');
|
||||
console.log('RESPONSE', resp);
|
||||
});
|
||||
},
|
||||
});
|
||||
|
||||
const backend = createBackend();
|
||||
backend.add(demoPlugin);
|
||||
```
|
||||
|
||||
This will be run every time you restart the backend. Note that after any change
|
||||
in the URL Reader code, you need to stop the backend and restart, since the
|
||||
`reader` instance is memoized and does not update on hot module reloading. Also,
|
||||
there are a lot of unit tests written for the URL Readers, which you can make
|
||||
use of.
|
||||
Reference in New Issue
Block a user