paulononato/backstage
1
0
Fork 0
Code Issues Pull Requests Actions 31 Packages Projects Releases Wiki Activity

Consolidate url reader docs

Browse Source 
Signed-off-by: Johan Haals <johan.haals@gmail.com>
This commit is contained in:
Johan Haals
2024-08-27 14:30:01 +02:00
parent 6853aac3e0
commit 1d06e061ab
2 changed files with 96 additions and 306 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plugins/url-reader.md
-306
View File
@@ -1,306 +0,0 @@
---
id: url-reader
title: URL Reader
sidebar_label: URL Reader
# prettier-ignore
description: URL Reader is a backend core service responsible for reading files from external locations.
---
## Concept
Some of the core plugins of Backstage have to read files from an external
location. [Software Catalog](../features/software-catalog/index.md) has to read
the [`catalog-info.yaml`](../features/software-catalog/descriptor-format.md)
entity descriptor files to register and track an entity.
[Software Templates](../features/software-templates/index.md) have to download
the template skeleton files before creating a new component.
[TechDocs](../features/techdocs/README.md) has to download the markdown source
files before generating a documentation site.
Since, the requirement for reading files is so essential for Backstage plugins,
the
[`coreServices.urlReader`](../reference/backend-plugin-api.coreservices.urlreader.md)
package provides a dedicated API for reading from such URL based remote
locations like GitHub, GitLab, Bitbucket, Google Cloud Storage, etc. This is
commonly referred to as "URL Reader". It takes care of making authenticated
requests to the remote host so that private files can be read securely. If users
have [GitHub App based authentication](../integrations/github/github-apps.md) set up, URL Reader even
refreshes the token, to avoid reaching the GitHub API rate limit.
As a result, plugin authors do not have to worry about any of these problems
when trying to read files.
## Interface
This service instance contains all the default URL Reader providers
in the backend-defaults package including GitHub, GitLab, Bitbucket, Azure, Google
GCS. As the need arises, more URL Readers are being written to support different
providers.
The generic interface of a URL Reader instance looks like this.
```ts
export interface UrlReaderService {
/**
* Reads a single file and return its content.
*/
readUrl(
url: string,
options?: UrlReaderServiceReadUrlOptions,
): Promise<UrlReaderServiceReadUrlResponse>;
/**
* Reads a full or partial file tree.
*/
readTree(
url: string,
options?: UrlReaderServiceReadTreeOptions,
): Promise<UrlReaderServiceReadTreeResponse>;
/**
* Searches for a file in a tree using a glob pattern.
*/
search(
url: string,
options?: UrlReaderServiceSearchOptions,
): Promise<UrlReaderServiceSearchResponse>;
}
```
## Using the URL Reader service inside a plugin
The following example shows how to get the URL Reader service in your `example` backend plugin to read a file and a directory from a GitHub repository.
```ts
import {
coreServices,
createBackendPlugin,
} from '@backstage/backend-plugin-api';
import os from 'os';
createBackendPlugin({
pluginId: 'example',
register(env) {
env.registerInit({
deps: {
urlReader: coreServices.urlReader,
},
async init({ urlReader }) {
const buffer = await urlReader
.read('https://github.com/backstage/backstage/blob/master/README.md')
.then(r => r.buffer());
const tmpDir = os.tmpdir();
const directory = await urlReader
.readTree(
'https://github.com/backstage/backstage/tree/master/packages/backend',
)
.then(tree => tree.dir({ targetDir: tmpDir }));
},
});
},
});
```
When any of the methods on this instance is called with a URL, URL Reader
extracts the host for that URL (e.g. `github.com`, `ghe.mycompany.com`, etc.).
Using the
[`@backstage/integration`](https://github.com/backstage/backstage/tree/master/packages/integration)
package, it looks inside the
[`integrations:`](https://github.com/backstage/backstage/blob/d5c83bb889b8142e343ebc4e4c0b90a02d1c1a3d/app-config.yaml#L134-L158)
config of the `app-config.yaml` to find out how to work with the host based on
the configs provided like authentication token, API base URL, etc.
Once the reader instance is available inside the plugin, one of its methods can
directly be used with a URL. Some example usages -
- [`readUrl`](https://github.com/backstage/backstage/blob/38f3827/plugins/catalog-backend/src/modules/codeowners/lib/read.ts#L24-L34) -
Catalog using the `readUrl` method to read the CODEOWNERS file in a repository.
- [`readTree`](https://github.com/backstage/backstage/blob/33ebb28/plugins/techdocs-node/src/helpers.ts#L147-L165) -
TechDocs using the `readTree` method to download markdown files in order to
generate the documentation site.
- [`readTree`](https://github.com/backstage/backstage/blob/33ebb28/plugins/techdocs-node/src/stages/prepare/url.ts#L55-L78) -
TechDocs using `NotModifiedError` to maintain cache and speed up and limit the
number of requests.
- [`search`](https://github.com/backstage/backstage/blob/38f3827/plugins/catalog-backend/src/modules/core/UrlReaderProcessor.ts#L120-L144) -
Catalog using the `search` method to find files for a location URL containing
a glob pattern.
Note that URL Readers which target git-based version control systems may, under
the hood, leverage the ability to create tar archives based on a specific git
commit-ish. A consequence of this is that files and directories configured with
[the `export-ignore` attribute](https://git-scm.com/docs/gitattributes#_creating_an_archive)
via `.gitattributes` will not be visible to the URL reader when using the
`readTree` or `search` methods.
Be aware of this limitation and ensure that end-users of your plugin are also
aware via, for example, documentation.
## Writing a new URL Reader
If the available URL Readers are not sufficient for your use case and you want
to add a new URL Reader for any other provider, you are most welcome to
contribute one!
Feel free to use the
[GitHub URL Reader service](https://github.com/backstage/backstage/blob/ce2ca68f07ad3334401d3277b989bf145b728a64/packages/backend-defaults/src/entrypoints/urlReader/lib/GithubUrlReader.ts#L60)
as a source of inspiration.
### 1. Add an integration
The provider for your new URL Reader can also be called an "integration" in
Backstage. The `integrations:` section of your Backstage `app-config.yaml`
config file is supposed to be the place where a Backstage integrator defines the
host URL for the integration, authentication details and other integration
related configurations.
The `@backstage/backend-defaults` package is where the URL Reader specific
code lives. Functions like "read the integrations config and process it",
"construct headers for authenticated requests to the host" or
"convert a plain file URL into its API URL for downloading the file"
would live in `@backstage/integrations` so that it is sharable across Backstage.
### 2. Create the URL Reader
Create a new class which implements the
[`UrlReaderService` type](https://github.com/backstage/backstage/blob/c4b8169/packages/backend-plugin-api/src/services/definitions/UrlReaderService.ts#L26)
inside `@backstage/backend-defaults`. Create and export a static `factory` method
which reads the integration config and returns a map of host URLs the new reader
should be used for. See the
[GitHub URL Reader](https://github.com/backstage/backstage/blob/ce2ca68f07ad3334401d3277b989bf145b728a64/packages/backend-defaults/src/entrypoints/urlReader/lib/GithubUrlReader.ts#L61-L73)
for example.
### 3. Implement the methods
We want to make sure all URL Readers behave in the same way. Hence if possible,
all the methods of the `UrlReaderService` interface should be implemented. However it
is okay to start by implementing just one of them and create issues for the
remaining.
#### `readUrl`
`readUrl` method expects a user-friendly URL, something which can be copied from
the browser naturally when a person is browsing the provider in their browser.
- ✅ Valid URL :
`https://github.com/backstage/backstage/blob/master/ADOPTERS.md`
- ❌ Not a valid URL :
`https://raw.githubusercontent.com/backstage/backstage/master/ADOPTERS.md`
- ❌ Not a valid URL : `https://github.com/backstage/backstage/ADOPTERS.md`
Upon receiving the URL, `readUrl` converts the user-friendly URL into an API URL
which can be used to request the provider's API.
`readUrl` then makes an authenticated request to the provider API and returns the response containing the file's contents and ETag(if the provider supports it).
#### `readTree`
`readTree` method also expects user-friendly URLs similar to `read` but the URL
should point to a tree (could be the root of a repository or even a
sub-directory).
- ✅ Valid URL : `https://github.com/backstage/backstage`
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master`
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/docs`
Using the provider's API documentation, find out an API endpoint which can be
used to download either a zip or a tarball. You can download the entire tree
(e.g. a repository) and filter out in case the user is expecting only a
sub-tree. But some APIs are smart enough to accept a path and return only a
sub-tree in the downloaded archive.
#### search
`search` method expects a glob pattern of a URL and returns a list of files
matching the query.
- ✅ Valid URL :
`https://github.com/backstage/backstage/blob/master/**/catalog-info.yaml`
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/**/*.md`
- ✅ Valid URL :
`https://github.com/backstage/backstage/blob/master/*/package.json`
- ✅ Valid URL : `https://github.com/backstage/backstage/blob/master/READM`
The core logic of `readTree` can be used here to extract all the files inside
the tree and return the files matching the pattern in the `url`.
### 4. Add to available URL Readers
There are two ways to make your new URL Reader available for use.
You can choose to make it open source, by updating the
[`default` factory](https://github.com/backstage/backstage/blob/ce2ca68f07ad3334401d3277b989bf145b728a64/packages/backend-defaults/src/entrypoints/urlReader/lib/UrlReaders.ts#L82-L102)
method of URL Readers.
But for something internal which you don't want to make open source, you can
update your `packages/backend/src/index.ts` file and update how the `reader`
instance is created.
```ts
// File: packages/backend/src/index.ts
import { urlReaderFactoriesServiceRef } from '@backstage/backend-defaults/urlReader';
import {
createBackendFeatureLoader,
createServiceFactory,
} from '@backstage/backend-plugin-api';
const customReader = createServiceFactory({
service: urlReaderFactoriesServiceRef,
deps: {},
async factory() {
return CustomUrlReader.factory;
},
});
const backend = createBackend();
// backend.add() of other plugins and modules excluded
backend.add(customReader);
```
### 5. Caching
All of the methods above support an ETag based caching. If the method is called
without an `etag`, the response contains an ETag of the resource (should ideally
forward the ETag returned by the provider). If the method is called with an
`etag`, it first compares the ETag and returns a `NotModifiedError` in case the
resource has not been modified. This approach is very similar to the actual
[`ETag`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag) and
[`If-None-Match`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match)
HTTP headers.
### 6. Debugging
When debugging one of the URL Readers, you can straightforward use the
[`reader` instance created](https://github.com/backstage/backstage/blob/ebbe91dbe79038a61d35cf6ed2d96e0e0d5a15f3/packages/backend/src/index.ts#L57)
when the backend starts and call one of the methods with your debugging URL.
```ts
// File: packages/backend/src/index.ts
import {
coreServices,
createBackendPlugin,
} from '@backstage/backend-plugin-api';
const demoPlugin = createBackendPlugin({
pluginId: 'demo',
register(env) {
env.registerInit({
deps: {
urlReader: coreServices.urlReader,
},
async init({ urlReader }) {
const resp = urlReader.read('http://my-url');
console.log('RESPONSE', resp);
});
},
});
const backend = createBackend();
backend.add(demoPlugin);
```
This will be run every time you restart the backend. Note that after any change
in the URL Reader code, you need to stop the backend and restart, since the
`reader` instance is memoized and does not update on hot module reloading. Also,
there are a lot of unit tests written for the URL Readers, which you can make
use of.